> ## Documentation Index
> Fetch the complete documentation index at: https://podonos.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Ranking
> Rank multiple models from best to worst.
## Intro
When comparing three or more speech synthesis models, a ranking evaluation is an effective method for determining the relative quality of each model. Rather than comparing pairs individually, evaluators listen to a set of audio samples generated from the same script and rank them from best to worst.
The Ranking evaluation is flexible in its evaluation criteria. You can rank models based on naturalness, overall preference, clarity, expressiveness, or any other quality dimension that matters to your use case.
* **Objective**: Determine the relative ordering of multiple models by having evaluators rank them.
* **Use Case**: Ideal for comparing TTS providers, model versions, or synthesis configurations side by side.
* **Type**: `RANKING` in the SDK, or `RANKING_REF` when every set should be judged against a reference.
## Example
In this example, we compare three different TTS providers by generating speech from the same scripts and submitting them for ranking evaluation. Here is a code example that you can immediately execute:
```python python theme={null}
import podonos
from podonos import *
import provider_a, provider_b, provider_c
client = podonos.init()
etor = client.create_evaluator(
name='TTS Provider Ranking',
desc='Ranking evaluation across TTS providers',
type='RANKING',
lan='en-us',
num_eval=10,
)
scripts = [
'But in less than five minutes',
'The two doctors therefore entered the room alone',
]
for script in scripts:
path_a = provider_a.synthesize(text=script, output='provider_a.wav')
path_b = provider_b.synthesize(text=script, output='provider_b.wav')
path_c = provider_c.synthesize(text=script, output='provider_c.wav')
etor.add_ranking_set([
File(path=path_a, model_tag='Provider_A', tags=['tts']),
File(path=path_b, model_tag='Provider_B', tags=['tts']),
File(path=path_c, model_tag='Provider_C', tags=['tts']),
])
etor.close()
```
Ok, let's go line by line.
Let's first create a new instance of `Client`.
```python python theme={null}
client = podonos.init()
```
Then, you create a new instance of `Evaluator` with `type='RANKING'`:
```python python theme={null}
etor = client.create_evaluator(
name='TTS Provider Ranking',
desc='Ranking evaluation across TTS providers',
type='RANKING',
lan='en-us',
num_eval=10,
)
```
For each script, generate speech from all providers and add a ranking set. Each ranking set contains one audio file per provider.
```python python theme={null}
etor.add_ranking_set([
File(path=path_a, model_tag='Provider_A', tags=['tts']),
File(path=path_b, model_tag='Provider_B', tags=['tts']),
File(path=path_c, model_tag='Provider_C', tags=['tts']),
])
```
Finally, close the `Evaluator` object.
```python python theme={null}
etor.close()
```
With this, you can rank multiple TTS providers via
[podonos](https://pypi.org/project/podonos/) from real human evaluators.
Once these steps finish, you can check the results in your [Workspace](https://workspace.podonos.com).
## Ranking Against a Reference
Sometimes "best" only means something relative to a target. If you are matching a
specific voice, restoring a recording, or reproducing a reference performance, the
question is not which sample sounds nicest but which one is **closest to the
reference**.
`RANKING_REF` adds exactly that: each ranking set carries one reference audio in
addition to the candidates. Evaluators hear the reference in every match of that
set, and it is never itself a candidate.
* **Objective**: Rank models by how closely they match a per-script reference.
* **Use Case**: Voice cloning, speech restoration, dubbing, any task with a target.
* **Type**: `RANKING_REF` in the SDK.
The code is the ranking example with one extra `File`:
```python python theme={null}
import podonos
from podonos import *
import provider_a, provider_b, provider_c
client = podonos.init()
etor = client.create_evaluator(
name='TTS Provider Ranking with Reference',
desc='Which provider is closest to the target voice?',
type='RANKING_REF',
lan='en-us',
num_eval=10,
)
scripts = [
'But in less than five minutes',
'The two doctors therefore entered the room alone',
]
for script in scripts:
path_ref = f'reference/{script[:20]}.wav'
path_a = provider_a.synthesize(text=script, output='provider_a.wav')
path_b = provider_b.synthesize(text=script, output='provider_b.wav')
path_c = provider_c.synthesize(text=script, output='provider_c.wav')
etor.add_ranking_set([
File(path=path_a, model_tag='Provider_A', tags=['tts']),
File(path=path_b, model_tag='Provider_B', tags=['tts']),
File(path=path_c, model_tag='Provider_C', tags=['tts']),
File(path=path_ref, model_tag='Reference', tags=['tts'], is_ref=True),
])
etor.close()
```
### Rules
Every set needs one `File` with `is_ref=True` and at least two candidates. A set
without a reference, or with two, is rejected when you call `add_ranking_set()`.
The SDK sorts it into position for you, so you can build the list however is
convenient. These are equivalent:
```python python theme={null}
etor.add_ranking_set([ref_file, file_a, file_b])
etor.add_ranking_set([file_a, file_b, ref_file])
```
The reference audio changes from script to script, but its `model_tag` must stay the
same across every set. It names the *role*, not the file. Using `Reference_1`,
`Reference_2`, ... makes your workspace treat each one as a separate model and splits
the reference into one row per tag.
```python python theme={null}
# do
File(path=ref_for_script_1, model_tag='Reference', is_ref=True)
File(path=ref_for_script_2, model_tag='Reference', is_ref=True)
# don't
File(path=ref_for_script_1, model_tag='Reference_1', is_ref=True)
File(path=ref_for_script_2, model_tag='Reference_2', is_ref=True)
```
`Reference` and `Provider_A` are fine together; a reference tagged `Provider_A`
is not.
Candidates are still ranked only against each other. The reference is a listening
aid, so it never appears in the Bradley-Terry scores and never shows up as a choice.
A set of N candidates produces N-1 matches whether or not a reference is present.
## Use Case
Consider a scenario where you are evaluating multiple TTS providers to decide which one to integrate into your product. Each provider may have different strengths. One might excel at naturalness while another handles proper nouns better. Using the Ranking evaluation, you can have human evaluators directly compare all providers on the same scripts and produce a clear ordering, giving you confidence in your selection.
## How It Works
Rankings are computed using the **[Bradley-Terry (BT) model](https://en.wikipedia.org/wiki/Bradley%E2%80%93Terry_model)**, a well-established statistical method for deriving global rankings from pairwise comparisons. Each pair of models is compared by human evaluators, and the aggregated results are used to estimate a score for every model via maximum likelihood estimation.
A key challenge in pairwise ranking is deciding **which pairs to compare**. With many models, the number of possible pairs grows quickly, but evaluation budgets are limited. Comparing pairs with obvious quality differences wastes valuable evaluations, while neglecting certain pairs leaves gaps in the data.
To address this, Podonos uses adaptive pairing based on **Fisher information**. Fisher information quantifies how much a given comparison will improve the overall ranking accuracy. Comparisons between similarly-ranked models yield the most information, while lopsided matchups yield little. The system dynamically balances **exploration** (ensuring all pairs are observed) and **exploitation** (focusing on the most informative pairs), adapting automatically as data accumulates throughout the evaluation.