Skip to main content

​
Intro

An effective way of evaluating subjective preferences between two pieces of speech or audio is the preference test. This evaluation helps to determine which of the two audio samples is more favored by evaluators. The Preference Test uses a scale to capture listener preferences, typically ranging from -2 to 2, with 1 granularity. The scale is as follows:
  • 2: A is strongly preferred
  • 1: A is preferred
  • 0: About the same
  • -1: B is preferred
  • -2: B is strongly preferred
Through the Preference Test, you can gather insights into which audio sample is more appealing to your audience, allowing for more informed decisions in audio production and refinement.
A and B are decided by model_tag, not by which argument you passed the file as. The SDK sorts the two stimuli by model_tag, so the model whose tag sorts first is always A. This holds across every group, which is what makes the aggregate score meaningful.You do not need to keep a consistent argument order, and you should not shuffle it yourself — each evaluator already hears the two samples in a randomized position. See Bias Minimization.
preference

​
Example

In this example, let’s assume you are developing a new speech synthesis model, named “myTTS”. You will compare your own model developed yesterady and the one developed today. Here is a code example that you can immediately execute:
Once the evaluation steps finish, you can check the preferences in your Workspace.