Five new language locales
Russian, Cantonese, Vietnamese, Greek, and Tagalog are now supported for evaluation — same panels, same metrics, native-speaker raters in-region:- Russian (Russia) —
ru-ru - Cantonese (Hong Kong) —
yue-hk - Vietnamese (Vietnam) —
vi-vn - Greek (Greece) —
el-gr - Tagalog (Philippines) —
fil-ph
Language checks before you pay
Podonos now checks every evaluator-facing text against the evaluation’s language before the evaluation starts: instructions, questions, answer options, and scale labels.- Setup checks on the summary step of evaluation setup quotes each text that isn’t in the evaluation’s language, with a link to edit the instructions or questions.
- Keep as is clears a finding when a bilingual setup is intended. Change the text afterward and the finding comes back.
- A question review reads the instruction and the questions together and lists what doesn’t fit, such as an instruction that promises a rating for each clip when no question asks about a single clip.
- Untranslated text stops the request before any charge, and the evaluation stays a draft. The message names the text to translate.
podonos 0.49.0, close(auto_start=True) raises EvaluationSetupError with every reason in the message. It subclasses HTTPError, so existing handlers still catch it. See close() in the SDK reference.Pay by monthly invoice
Postpay billing is now available. Evaluations start without a prepaid balance, and each month’s usage arrives on one invoice with 30-day payment terms.- Workspace admins switch between the prepaid balance and Postpay (monthly invoice) on the Billing & Payments page.
- A switch to postpay applies at once. A switch back applies at once when no postpay charges are open, and otherwise on the 1st of the next month (UTC).
Ranking ties reported as TIE
A ranking match scored as a draw now reads "winner": "TIE" in both JSON downloads. The tournament still advances on a seeded tie-break, but the export no longer reports that pick as a win. Rank scores are unchanged, since they already skip draws.Run evaluations end to end from the SDK
A pipeline can now create, start, and track an evaluation without opening the web app.auto_start=Truestarts the evaluation fromclose()once uploaded files finish processing, and bills your workspace: the prepaid balance, or the monthly invoice on postpay.start_timeoutcaps the wait, 1800 seconds by default.get_evaluation_progress(evaluation_id)reads one evaluation’s status and progress.get_evaluation_list()now returnsprogress,started_time, andended_timeon every row.resume_evaluator()resumes an interrupted upload with an API key.- A start refused for insufficient balance emails every workspace member and posts to your connected Slack channel, so the person who can top up hears about it, not only the CI log. Alerts go out at most once every 6 hours per workspace.
status == "COMPLETED" as done. Progress tops out at 90 until then. Requires podonos 0.47.0 or later. See the SDK reference.Pick your own evaluation emails
Each member now chooses which evaluation emails they receive in each workspace: Evaluation started, Evaluation completed, and Evaluation canceled. Find them under Settings → Notifications → Your Email Notifications.The workspace email switch still comes first. When an admin turns it off, no member gets these emails. Balance and payment emails can’t be muted per member.Card digits in payment lists
Saved cards now show their last four digits wherever you pick one: at checkout, when changing the recurring payment method, and when setting the default card. Two cards from the same brand no longer look alike.Four more Indian languages
Bengali, Gujarati, Marathi, and Telugu are now supported for evaluation — same panels, same metrics, native-speaker raters in-region:- Bengali (India) —
bn-in - Gujarati (India) —
gu-in - Marathi (India) —
mr-in - Telugu (India) —
te-in
Rank against a reference
RANKING_REF ranks models by how close each one comes to a reference clip. Each ranking set carries one reference, which plays in every match of its set and never competes. Where CSMOS compares two models against a reference, RANKING_REF takes every model in the set.- Pass the reference to
add_ranking_set()as aFilewithis_ref=True, alongside at least two candidates. - Templates support it too, in the web app and through
create_evaluator_from_template().
podonos 0.45.0 or later. See Ranking.Confirm your Slack connection
Connecting Slack now posts a welcome message to the channel, listing the notifications it will receive. If that message can’t be delivered, a banner says so on the spot, instead of the first real notification going missing days later.Workspace admins can check the connection again at any time with Send test message under Settings → Notifications, for example after the channel is deleted.One position per model in comparisons
In comparative evaluations, the SDK now gives eachmodel_tag the same position in every group, whatever order you pass files to add_files(). Model A stays the same model in every trial. CMOS and DMOS store the reference last, as CSMOS already did.The platform refuses an inconsistent order at upload, not at checkout after the whole upload has finished.The minimum supported SDK version is now 0.44.0. Upgrade with pip install -U podonos.Annotation votes in JSON downloads
Annotation questions with options now export their votes. Download JSON includes each question’s option distribution, and Download JSON + response includes the option each evaluator picked.Four new language locales
Below four language locales are live for evaluation with same panels, same metrics, native-speaker raters in-region.- Sinhala (Sri Lanka) —
si-lk - Arabic-Egypt —
ar-eg - Arabic-UAE —
ar-ae - Arabic-Saudi Arabia —
ar-sa
Dutch and Swedish
Two more European locales are live for evaluation. Expected turnaround is 48 hours for both.- Dutch (Netherlands):
nl-nl - Swedish (Sweden):
sv-se
podonos 0.43.0 or later. See the language list.A public language reference
Every supported locale, on one page — with the expected panel turnaround time for each. Bookmark it before your next project.See the language list →
SOC 2 Type II — completed
Podonos completed its SOC 2 Type II audit. Controls covering security, availability, and confidentiality were tested across the audit window and operated effectively.The report sits on the trust center.trust.podonos.com →
Annotations in JSON downloads
Download JSON and Download JSON + response now carry span annotations, and the per-evaluator download includes each span’s content and reason. Annotation-only evaluations, which used to download empty, now export in full.Uploads that hold up under retries
Interrupted SDK uploads now resume reliably, and a retried or resumed upload registers each file exactly once. Duplicate registrations used to surface later as a group size mismatch. Usepodonos 0.42.0 or later.Evaluation reliability — what runs behind every panel
Reliable scores don’t happen by accident.Behind every panel: vetted evaluator pools, per-session acoustic environment checks, embedded attention tests, audio shuffling and loudness normalization, and post-session cohort scoring with automatic backfill of underperformers.The full pipeline, what we screen for, what we reject, and how scores stay comparable run to run, language to language is documented.
Expanded Language Coverage: South Asian Languages
We have added support for four major South Asian languages, broadening our reach for voice applications targeting the Indian subcontinent:- Hindi (India) —
hi-in - Tamil (India) —
ta-in - Kannada (India) —
kn-in - Malayalam (India) —
ml-in
Mandatory Annotations for Specific Scores
You can now require evaluators to provide an annotation when they select certain score values.For example, if an evaluator rates a sample as “2”, they must explain why — ensuring you always get qualitative context behind quantitative ratings.- Define which score values trigger a mandatory annotation
- Annotation requirements are set per question, giving you precise control
- Works across single, double, and triple stimulus evaluation types
Automatic Annotation Translation
Understanding evaluator feedback just got easier — regardless of the language it was written in.Previously, annotations were only available in the language the evaluator used. Now, annotations are automatically translated into English in the Report views. Run evaluations with native-language evaluators worldwide and read every annotation without extra steps.
Improved Audio Download Reliability
We resolved an issue that caused “Failed to fetch” errors when downloading audio files from evaluations. This was caused by a cross-origin configuration issue affecting certain file access patterns.Audio file downloads now work reliably across all evaluation types and browser configurations. No action is needed on your end.
Evaluator Preview
You can now preview what evaluators will see before launching an evaluation.This preview shows:- Instructions and questions as they appear to evaluators
- The actual layout evaluators interact with
Richer Annotation Instructions
Annotation just became more expressive. You can now add an annotation type directly to each question and include a detailed description explaining exactly what should be annotated.For example, you can explicitly ask evaluators to annotate:- Incorrect pronunciation
- Background noise
- Awkward or unnatural pronunciation segments
Comparative Similarity SDK Workflow
Creating comparative similarity evaluations via SDK is now simpler and more consistent.Previously, evaluations had to be partially created in the SDK and then manually completed in the workspace. Now, you can use atemplate.json directly in the SDK to create a complete evaluation end-to-end without manual workspace setup, making evaluation creation more reproducible and automation-friendly.
Passwordless Login
As an additional security step, we have fully transitioned to magic URL login:- No more passwords to remember or manage
- All existing passwords have been securely deleted from our system
SOC 2 Type I Certification
Podonos has passed SOC 2 Type I and has successfully completed external penetration testing. We are currently in the observation period for Type II.You can always find our latest security updates at our Trust Center.
Slack Notifications for Evaluation Status
Stay updated without checking your inbox.In addition to email notifications, Podonos now supports Slack integration for evaluation status updates. You can receive notifications when an evaluation starts and when it finishes, directly in Slack.- Slack alerts for evaluation start and completion
- Ideal for team visibility and real-time monitoring
- Easy to integrate into existing workflows
Advanced Tag Filtering in Deep Analysis
We added an Any/All toggle to the tag filter within the Deep Analysis section of reports.This feature provides flexible filtering logic:- Any: Matches results containing at least one of the selected tags.
- All: Matches results containing every selected tag.
Smarter Tag Control with Invisible Tags
Keep reports clean without losing important context.You can now hide specific tags from your report pages while keeping them intact in the evaluation setup. This makes it easier to share reports externally without exposing internal or experimental metadata.- Hide selected tag(s) from the report UI
- Tags remain available internally for filtering and organization
- Configure hidden tags in Report Settings → Hide tags
Auto Translation During Evaluation Setup
Create multilingual evaluations with less effort.When creating an evaluation, you first select a target language. Any instruction or question you enter, regardless of the original language, can be automatically translated into the target language with a single click.- Target language defined at evaluation creation
- One-click translation for instructions and questions
- Reduces friction when setting up global evaluations
Filter by demographic
We have added a feature that allows you to filter evaluation results by combining tags with evaluator demographics, such as region, age, and gender. You can create up to 10 different combinations to compare results at a glance. Check it out in the Sample Report!
Status Page
We have released our Status Page, enabling real-time monitoring of system status. This includes service uptime, server availability, and API health, providing greater transparency and reliability for your operations.
Trust Center
We have officially initiated our SOC 2 Type II audit. Security and data protection are core foundations at Podonos, and this audit represents another step toward meeting the highest industry standards. We are committed to transparent operations and maintaining a secure environment for all customer data.To learn more about our security practices or stay updated on our audit progress, please visit our Trust Center.
Save and Revert Buttons
You can now manually Save your progress or Revert changes while creating an evaluation. These controls provide greater flexibility and safety when configuring your evaluation tasks.Report settings updates
You no longer need to wait for an evaluation to complete to access report settings. You can now modify report configurations—including title, description, sharing setup, and short URL—anytime, even while the evaluation is in progress.Quick Access to Previous Recipients
In Report Sharing settings, you can now quickly access and add emails or domains used in previous evaluations. This list history feature streamlines the process of sharing reports with your frequent stakeholders.Expanded Language Coverage
We have expanded our language support to include key regional dialects:- Indian English
- Canadian French
- Brazilian Portuguese and Portugal Portuguese
Podonos Flash (Alpha): The World’s First Voice Evaluation API
Introducing Podonos Flash, the first-ever API designed for real-time voice evaluation.While human evaluation offers superior accuracy, it can be resource-intensive. Podonos Flash bridges this gap by instantly assessing naturalness and noise quality with human-like precision, enabling large-scale performance monitoring and optimization.Currently in Alpha Testing. We are refining the system for a full official launch in January 2026.During this Alpha phase, we are offering limited early access to select partners. To request access, please contact us at hello@podonos.com.Learn more
Workspace Enhancements
We have introduced key administrative and billing improvements to the Workspace:- Admin Role Transfer: Seamlessly transfer administrative privileges within your organization.
- Multi-Payment Method Support: Register and manage multiple payment methods for flexible billing operations.
New: Transcription Services
We have launched High-Accuracy Transcription Services, essential for benchmarking speech-to-text models and generating high-quality training datasets.Our service supports 12 major languages, including English, Spanish, French, Japanese, Korean, and German, ensuring reliable ground truth data for your AI models.
U.S. Address Speech Dataset
Accurate address processing is critical for voice AI systems yet remains a common failure point. To address this, we have curated a Verified U.S. Address Speech Dataset to help benchmark and improve your model’s performance on complex address formats.Contact us to access this high-value dataset.
Secure Report Sharing
Enhance team collaboration with Secure Report Sharing. You can now securely distribute evaluation reports to internal teammates and external partners, ensuring stakeholders have controlled access to performance insights.
Recommended Templates
Streamline your evaluation workflow with Recommended Templates.Available via both SDK and Web, these pre-configured templates allow you to rapidly set up evaluations aligned with standard industry goals, reducing configuration time and ensuring best practices.
Wizard
We’ve introduced a Wizard to streamline evaluation creation and accelerate your workflow.The API-based approach revolutionizes the evaluation process by enabling rapid evaluation creation through just a few clicks, eliminating the need for manual audio file generation and upload procedures. This streamlined method significantly reduces setup time while maintaining the same comprehensive evaluation capabilities.Currently, the API-based evaluation method supports single audio evaluation for naturalness and quality evaluation and double audio evaluation for voice similarity and preferences evaluation. We are actively working to extend API-based support to additional evaluation types and will be rolling out these capabilities in upcoming releases.
Automatic Loudness Normalization
All audio inputs are automatically normalized to -21 LUFS before presentation to human evaluators, ensuring consistent and accurate evaluation results regardless of original audio levels.Public Report Setup
You can now customize your report titles and descriptions at any time. When publishing reports publicly, you have full control over audio playability settings, model names, and tags.
Data Collection (Beta)
Multi-language voice data collection from diverse global contributors to improve model performance. This service is currently available to selected early customers who are seeing significant improvements in model accuracy and speed.Markdown Descriptions
Add comprehensive descriptions for your models and comparisons using markdown formatting, similar to README.md files on GitHub or Hugging Face. This enables rich documentation with formatted text, links, and structured content.
Report Download
Download complete analysis data in JSON format with three different organizational structures to suit your specific needs and workflows.
Voice Cloning Evaluation
Comprehensive voice similarity measurement for voice cloning applications. Our system supports three-way similarity analysis comparing a reference voice with two target outputs, enabling precise evaluation of voice cloning quality.
Model-related questions
In three-way similarity analysis, you can now ask specific questions related to each individual model. Through model-related questions, you can conduct separate evaluations for each model within the triple audio evaluation framework, providing the same level of detailed evaluation as single audio evaluations.This feature enables comprehensive analysis by allowing evaluators to focus on specific aspects of each model’s performance while maintaining the comparative context of the three-way evaluation structure.
Collaborate with team
Enable teamwork by sharing evaluations, feedback, and reports. Foster collaboration and ensure everyone stays aligned on goals and outcomes
Web-based evaluation
You don’t need an SDK to create an evaluation any more.
Evaluation template support
You create the template once, use it multiple times. Once you create your own template, you can re-use them in the following evaluations.New Billing & Payment
We have a new look in the Billing & Payments. You can set up the recurring payments and get additional discounts.High Scalability
You can evaluate up to 10k queries with 10k files in a single session.Workspace is revamped
Now, the workspace has a new look and a more organized menu.
Fully customize your own evaluations
So far, you can use the provided evaluation templates. Now, you can add your own messages and followup questions up to 8. Instead of asking a single question, you can ask multiple questions on different types.Add audio examples
Evaluator may wonder what is excellent and what is bad. You can add audio examples, so your evaluation will be more consistent.New format support
We now support FLAC type too in addition to wav and mp3.New evaluation report
We revamped the whole report presentation. With the new concept ofmodel_tag, you can group the evaulations into models and multiple tags and analyze more details.
Reasoning behind the rating
One of the core questions behind the report is that “why did the evaluator think so?”.So we added this new feature to ask the evaluators to annotate the words/phrases and put reasons behind the evaluation ratings.
6x faster upload in SDK
We previously made the file upload 2x faster. Again, we have made 6x faster upload, total 12x faster than last month.It can be faster depending on your internet speed.SDK documentation updated
We have revamped the whole SDK documents with more detailed descriptions and examples.Many more stability and bug fixes
We have fixed multiple performance and stability issues. Also fixed many bugs for more accurate evaluations.Custom question and detailed description
You can customize the question and the detailed description - So you can ask your own question to the evaluators instead of ours.2x faster file upload
Our backend service has become faster and the SDK uploads your files to evaluate in multiple upload sessions.Add the API key to the environment variable
See this document for details.In addition, we have tons of ideas around evaluation management, reliability, modality, speed improvement, and so on. If you have cool ideas, please feel free to let us know at hello@podonos.com.