What is subjective audio evaluation?

Subjective audio evaluation is the assessment of generated (e.g., by using generative AI models), processed (noise reduction, compression, echo cancellation, and so on) audio or speech by human listeners. The human evaluators play a crucial role in determining the effectiveness and quality of the audio output.

The main goals of subjective audio evaluation include:

  1. Naturalness: How closely does the AI-generated voice resemble a human voice?
  2. Quality: What is the overall quality level of the audio or speech? How much noise do you hear?
  3. Similarity: How similar is the AI-generated speech to the intended target or original speech? How similar is the voice of the French-speaking Elon Musk to his original English-speaking voice?
  4. Preferences: What are the listeners' preferences among different versions of AI-generated audio?

The ultimate goal is to gain insights into the usability of the output, and to find ways to further improve generative AI models, speech enhancement techniques, noise reduction algorithms, and other related technologies.

However, is this process simple? In reality, it is not unfortunately. Before executing such evaluations, you need to address numerous preliminary questions:

- And many more logistical and methodological considerations.

Assuming that you have relevant answers to most of the questions, let’s delve deeper into the evaluation types and scales. There are established subjective evaluation standards, recommended by the International Telecommunication Union (ITU) such ITU-T P.835, ITU-R BS.1534 (MUSHRA), and de facto standards used within the industry.

Assume you want to evaluate how natural the AI-generated human voice is. One widely used evaluation method for this type of assessment is the Mean Opinion Score (MOS). In this method, evaluators rate the naturalness of the audio on a scale from excellent, good, fair, poor, to bad. For statistically meaningful results, you must:

Typically, you would evaluate multiple audio files containing generated speech. By compiling all the data, you can compute overall naturalness statistics and create a table such as:

Voilà! Now you can draw a first insight into how natural the output speech is.

This example, however, overlooks many sophisticated details as the actual evaluation process is much more complex and cumbersome. Evaluations must consider the diversity of listeners, potential biases, the context in which the audio will be used, and the specific attributes of speech quality and naturalness that are most relevant to the application at hand.

In conclusion, subjective audio evaluation is a critical process in the development and refinement of AI-generated audio technologies. By thoroughly planning and executing these evaluations, we can gather valuable insights that drive the improvement of AI models and audio processing techniques, ultimately leading to more natural and high-quality audio experiences.


Other readings

Audio deepfake detection benchmark: Neural Defend leads as 13 systems clear 95%

Five new entries later, our audio deepfake detection benchmark has 28 systems, Neural Defend leads at 99.93%, and 13 systems now clear 95% accuracy.

|7 min read

Audio deepfake detection benchmark: deetech.ai takes the top spot

Five new systems later, our audio deepfake detection benchmark has grown to 23 entries, and eight systems now clear 95% accuracy.

|11 min read

Audio deepfake detection benchmark: an open model finally clears the production bar

When we last published this benchmark, four commercial systems sat above 95% accuracy and none of nine downloadable open-source models scored above 63%.

|10 min read

Audio Deepfake Detection Benchmark: the production bar moves to four

Nine new systems later, the top of our audio deepfake detection benchmark is crowded, the open-source gap is wider than ever, and speed sets systems apart.

|7 min read

Natural ≠ Preferred: What Our TTS Rankings Revealed About How Humans Actually Judge AI Voices

Naturalness and preference diverge: our US English TTS benchmark ranked Deepgram Aura 2 #1 for naturalness and Rime Arcana v3 #1 for preference.

|7 min read

Automatic Deepfake Audio Detection Benchmark

Every audio deepfake vendor reports accuracy around 99%, but none of those numbers are comparable: each comes from a test set the vendor controls.

|7 min read

Announcing the Selected Teams for the Podonos Research Support Program

After meeting speech and audio researchers and reviewing their applications, we announce the teams selected for the Podonos Research Support Program.

|2 min read

Benchmarking Chatterbox Turbo: How Resemble AI Evaluated Open-Source Voice AI with Podonos

Resemble AI's open-source Chatterbox showed open TTS models are viable alternatives to proprietary systems. Chatterbox Turbo takes that a step further.

|5 min read

Introducing Podonos Flash

Human evaluation gives voice AI teams the best insight but takes time, and fast AI evaluators often miss human perception. Podonos Flash changes that.

|4 min read

Podonos just raised $2.4M in pre-seed funding

Podonos has raised $2.4M in pre-seed funding to tackle voice AI evaluation, led by Serac Ventures with participation from Naver D2SF and KAIST Ventures.

|3 min read

Ready to unlockthe potential ofyour Voice AI Model?

Improve your model with trust