Reference

Podonos

This is a base module. You can import:

python

import podonos
from podonos import *

init()

Initialize the module and return an instance of Client.

string

API key you obtained from the workspace. For details, see the Get API key. If this is not set, the package tries to read PODONOS_API_KEY from the environment variable. Throws an error if both of them are not available.

string

default:"https://prod.podonosapi.com"

API base URL. You usually do not need to set this unless you are using a private, staging, or development API endpoint.

Returns an instance of Client.

python

client = podonos.init(api_key="<API_KEY>")

# Optional: pass the production API endpoint explicitly
client = podonos.init(api_key="<API_KEY>", api_url="https://prod.podonosapi.com")

Client

Client manages one or more Evaluator instances and evaluation history.

create_evaluator()

Create a new instance of Evaluator. One evaluator supports a single type of evaluation throughout its life cycle. If you want multiple types of evaluation, create multiple evaluators by calling create_evaluator() multiple times.

string

Name of this evaluation session. If empty, a random name is automatically generated and used.

string

Description of this evaluation session. This field is for your records, so later you can see how you generated the output files or trained your model.

string

default:"NMOS"

Evaluation type. One of the following:

Type	Description
`NMOS`	Naturalness Mean Opinion Score
`QMOS`	Quality Mean Opinion Score
`CMOS`	Reference-based comparison (compare target audio against reference)
`CSMOS`	Comparative similarity test between two target audios and one reference
`SMOS`	Similarity Mean Opinion Score
`P808`	Speech quality by ITU-T P.808
`PREF`	Preference test between two audios/speeches
`CUSTOM_SINGLE`	Custom single-stimulus evaluation
`CUSTOM_DOUBLE`	Custom double-stimulus evaluation
`RANKING`	Ranking evaluation across multiple stimuli

string

default:"en-us"

Specific language and locale of the speech. Currently we support:

Code	Description
`en-us`	English (United States)
`en-gb`	English (United Kingdom)
`en-au`	English (Australia)
`en-ca`	English (Canada)
`en-in`	English (India)
`ko-kr`	Korean (Korea)
`zh-cn`	Mandarin (China)
`es-es`	Spanish (Spain)
`es-mx`	Spanish (Mexico)
`fr-fr`	French (France)
`fr-ca`	French (Canada)
`de-de`	German (Germany)
`ja-jp`	Japanese (Japan)
`it-it`	Italian (Italy)
`pl-pl`	Polish (Poland)
`pt-pt`	Portuguese (Portugal)
`pt-br`	Portuguese (Brazil)
`id-id`	Indonesian (Indonesia)
`nl-nl`	Dutch (Netherlands)
`sv-se`	Swedish (Sweden)
`hi-in`	Hindi (India)
`ta-in`	Tamil (India)
`kn-in`	Kannada (India)
`ml-in`	Malayalam (India)
`si-lk`	Sinhala (Sri Lanka)
`ar-eg`	Arabic (Egypt)
`ar-ae`	Arabic (UAE)
`ar-sa`	Arabic (Saudi Arabia)
`audio`	General audio file

We will add more soon. Please check later again.

float

default:"1.0"

Granularity of evaluation scales. Supported values are 1.0 and 0.5.

int

default:"10"

Number of evaluations per sample. For example, if this is 10 for NMOS type evaluation, each audio file will be assigned to 10 humans, and the statistics of the evaluation output will be computed and presented in the final report.

int

default:"12"

Expected due time of the final report in hours. Must be at least 12. Depending on the hours, the pricing may change.

bool

default:"False"

Enable annotation to collect free-form text feedback from evaluators.

bool

default:"True"

Enable loudness normalization to ensure consistent audio volume levels during evaluation.

bool

default:"False"

If True, the evaluation automatically starts when you finish uploading the files. If False, you go to Workspace, confirm the evaluation session, and manually start the evaluation.

int

default:"20"

Maximum number of upload worker threads. If you experience a slow upload, please increase the number of workers.

int

default:"100"

Batch size for file verification API calls. Must be between 1 and 1000.

tuple[float, float]

default:"(5, 30)"

Timeout tuple for API requests, in seconds.

tuple[float, float]

default:"(5, 120)"

Timeout tuple for file verification requests, in seconds.

tuple[float, float]

default:"(10, 300)"

Timeout tuple for direct file upload requests, in seconds.

bool

default:"False"

Enable SDK-local upload ledger/resume support. The ledger records upload progress and the original evaluation contract so interrupted uploads can be resumed safely.

string

default:"None"

Optional SQLite upload ledger path when resume_upload=True. Use the same path with resume_evaluator() if the upload is interrupted.

Returns an instance of Evaluator.

etor = client.create_evaluator()

etor = client.create_evaluator(
    name="sim_iter100",
    desc="new_model_vs_sota",
    type="SMOS",
)

etor = client.create_evaluator(
    name="sim_iter100",
    desc="new_model_vs_sota",
    type="SMOS",
    lan="en-us",
    granularity=1.0,
    num_eval=100,
    due_hours=24,
    use_loudness_normalization=True,
    auto_start=True,
    verify_batch_size=100,
    resume_upload=True,
    upload_state_path="./podonos-upload-ledger.sqlite3",
)

resume_evaluator()

Resume uploads for an existing evaluation using an SDK upload ledger. Add files again in the same order as the interrupted run so the ledger can recover each file’s remote object identity.

Resume is safety-checked against the local ledger. The ledger must contain the original evaluation contract and session configuration for evaluation_id; otherwise the SDK refuses to resume. The SDK restores the original session details from the ledger where available, so do not use resume_evaluator() to change the evaluation type, language, template, batch size, or file identities.During resume, completed ledger rows are reused only when the current local file still matches the recorded path, content hash, size, and upload manifest. If a completed metadata or verification row no longer matches the local file, start a fresh evaluation or remove the stale ledger row before resuming.

string

required

Evaluation ID to resume.

string

required

Path to the SDK upload ledger created by a previous upload run. It must contain the original evaluation contract for evaluation_id.

string

Optional session name.

string

Optional session description.

string

default:"NMOS"

Evaluation type. Uses the same supported values as create_evaluator().

string

default:"en-us"

Language code. Uses the same supported values as create_evaluator().

float

default:"1.0"

Granularity of evaluation scales.

int

default:"10"

Number of evaluations per sample.

int

default:"12"

Expected due time of the final report in hours.

bool

default:"False"

Enable annotation to collect free-form text feedback from evaluators.

bool

default:"True"

Enable loudness normalization for evaluation.

bool

default:"False"

If True, the evaluation automatically starts after files are uploaded.

int

default:"20"

Maximum number of upload worker threads.

int

default:"100"

Batch size for file verification API calls.

tuple[float, float]

default:"(5, 30)"

Timeout tuple for API requests, in seconds.

tuple[float, float]

default:"(5, 120)"

Timeout tuple for file verification requests, in seconds.

tuple[float, float]

default:"(10, 300)"

Timeout tuple for direct file upload requests, in seconds.

python

etor = client.resume_evaluator(
    evaluation_id="<EVALUATION_ID>",
    upload_state_path="./podonos-upload-ledger.sqlite3",
)

create_evaluator_from_template()

When you create an evaluation using a template, all the questions and options defined in the template are automatically assigned to the new evaluation. This ensures consistency and saves time by reusing pre-defined content.

string

required

Name of this evaluation session. Required and must be non-empty.

string

required

The unique identifier of the template to base the new evaluation on.

string

Description of this evaluation session. This field is for your records.

int

default:"10"

Number of evaluations per sample.

bool

default:"False"

Enable annotation to collect free-form text feedback from evaluators. Cannot be used together with an annotations array in template JSON.

bool

default:"True"

Enable loudness normalization to ensure consistent audio volume levels during evaluation.

bool

default:"False"

If True, the evaluation automatically starts when you finish uploading the files.

int

default:"20"

Maximum number of upload worker threads.

int

default:"100"

Batch size for file verification API calls.

tuple[float, float]

default:"(5, 30)"

Timeout tuple for API requests, in seconds.

tuple[float, float]

default:"(5, 120)"

Timeout tuple for file verification requests, in seconds.

tuple[float, float]

default:"(10, 300)"

Timeout tuple for direct file upload requests, in seconds.

bool

default:"False"

Enable SDK-local upload ledger/resume support. The ledger records upload progress and the original evaluation contract so interrupted uploads can be resumed safely.

string

default:"None"

Optional SQLite upload ledger path when resume_upload=True. Use the same path with resume_evaluator() if the upload is interrupted.

etor = client.create_evaluator_from_template(
    name="Voice naturalness evaluation",
    desc="new_model_vs_competitor_model",
    num_eval=10,
    template_id="abcdef",
    use_loudness_normalization=True,
)

create_evaluator_from_template_json()

Create a new evaluation using a JSON template. This allows you to define custom evaluation structures programmatically.

Dict

Template JSON as a dictionary. Optional if json_file is provided.

string

Path to the JSON template file. Optional if json is provided.

string

Name of this evaluation session. Optional; if omitted, the SDK generates a name.

string | CustomType

default:"SINGLE"

Type of evaluation. Accepts either a string or CustomType enum value.

Value	Description	batch_size	File Configuration
`SINGLE`	Single stimulus evaluation	1	1 stimulus
`DOUBLE`	Double stimulus evaluation	2	2 stimuli (no reference)
`SINGLE_REF`	Reference-based comparison (CMOS style)	2	1 reference + 1 stimulus
`RANKING`	Ranking evaluation	2+	Multiple stimuli

You can use string values ("SINGLE", "DOUBLE", "SINGLE_REF", "RANKING") or the CustomType enum from podonos.common.enum:

from podonos.common.enum import CustomType

custom_type=CustomType.SINGLE_REF
custom_type="SINGLE_REF"

string

Description of this evaluation session. Optional.

string

default:"en-us"

Language for evaluation. See supported languages in create_evaluator().

int

default:"10"

Number of evaluations per sample.

bool

default:"False"

Enable annotation to collect free-form text feedback from evaluators. Cannot be used together with an annotations array in template JSON. When using custom annotation questions, define them in the annotations array instead.

bool

default:"True"

Enable loudness normalization to ensure consistent audio volume levels during evaluation.

bool

default:"False"

If True, the evaluation automatically starts when you finish uploading the files.

int

default:"20"

Maximum number of upload workers. Must be a positive integer.

int

default:"100"

Batch size for file verification API calls.

tuple[float, float]

default:"(5, 30)"

Timeout tuple for API requests, in seconds.

tuple[float, float]

default:"(5, 120)"

Timeout tuple for file verification requests, in seconds.

tuple[float, float]

default:"(10, 300)"

Timeout tuple for direct file upload requests, in seconds.

bool

default:"False"

Enable SDK-local upload ledger/resume support. The ledger records upload progress and the original evaluation contract so interrupted uploads can be resumed safely.

string

default:"None"

Optional SQLite upload ledger path when resume_upload=True. Use the same path with resume_evaluator() if the upload is interrupted.

# Using JSON dictionary
template = {
    "questions": [
        {
            "type": "SCORED",
            "question": "How natural is the voice?",
            "description": "Rate the quality of the voice",
            "options": [
                {"label_text": "Excellent"},
                {"label_text": "Good"},
                {"label_text": "Fair"},
                {"label_text": "Poor"},
                {"label_text": "Bad"},
            ],
        }
    ]
}

evaluator = client.create_evaluator_from_template_json(
    json=template,
    name="Quality Test",
    custom_type="SINGLE",
)

# Using JSON file
evaluator = client.create_evaluator_from_template_json(
    json_file="./template.json",
    name="Quality Test",
    custom_type="DOUBLE",
)

from podonos import File
from podonos.common.enum import CustomType

# CMOS-style reference comparison
template = {
    "questions": [
        {
            "type": "SCORED",
            "question": "How similar is the synthesized audio to the reference?",
            "options": [
                {"label_text": "Identical"},
                {"label_text": "Very similar"},
                {"label_text": "Somewhat similar"},
                {"label_text": "Different"},
                {"label_text": "Completely different"},
            ],
        }
    ]
}

evaluator = client.create_evaluator_from_template_json(
    json=template,
    name="CMOS Style Evaluation",
    custom_type=CustomType.SINGLE_REF,  # or "SINGLE_REF"
)

# Add files: 1 stimulus + 1 reference
evaluator.add_files(
    file0=File(path="synthesized.wav", model_tag="tts"),
    file1=File(path="reference.wav", model_tag="human", is_ref=True),
)

Returns an instance of Evaluator. Here’s the JSON template for reference:

Question
Option
Anchor Label
Instruction
Annotation

Question: Represents the main question posed to evaluators about the audio being assessed. It guides evaluators on the specific aspect of the audio they should focus on during the evaluation.

Parameter	Description	Required	Notes
`type`	Type of question. Options: `SCORED`, `NON_SCORED`, `COMPARISON`	Yes	Determines the structure and requirements of the question
`question`	The main question text	Yes	Must be provided for all question types
`description`	Additional details or context for the question	No	Optional for all question types
`options`	List of possible options. Only for `SCORED` and `NON_SCORED` types	Conditional	Must have between 1 and 9 options for `SCORED` and `NON_SCORED` types
`scale`	Scale for comparison. Only for `COMPARISON` type	Conditional	Must be an integer between 2 and 9 for `COMPARISON` type
`allow_multiple`	Allows multiple selections. Only for `NON_SCORED` type	Yes for `NON_SCORED`	Enables multiple choice selection
`has_other`	Includes an “Other” option. Only for `NON_SCORED` type	No	Adds an option for evaluators to specify an unlisted choice
`has_none`	Includes a “None” option. Only for `NON_SCORED` type	No	Adds an option for evaluators to select none of the listed choices
`related_model`	Related model for the question. Only for `Double` Evaluation type.	Conditional	Select which model the question is related to.
`anchor_label`	Labels for the ends of the comparison scale. Only for `COMPARISON` type.	Conditional	Provides context for what each end of the scale represents.

Important Notes:

SCORED and NON_SCORED questions can have a maximum of 9 options.
NON_SCORED questions must specify allow_multiple.
COMPARISON type questions must have a scale between 2 and 9.
related_model consists of ALL, MODEL_A and MODEL_B. Default is ALL. The related_model is only used for the question (not for instructions).

Option: Represents a possible answer or choice in a question. It provides evaluators with a range of options to choose from.

Parameter	Description	Required	Notes
`label_text`	The text displayed for the option	Yes	This is the text shown to the evaluator
`reference_file`	A reference audio file for the option. It helps the evaluator understand the quality of the audio	No	This file serves as a benchmark and is saved in the evaluation results

For options in SCORED and NON_SCORED questions:

score is automatically generated only for SCORED questions. If there are 5 options, the first option in the list receives a score of 5, the second option receives a score of 4, and so on, down to a score of 1 for the last option.
order is the index of the option in the list, starting from 0.

Anchor Label: Only for COMPARISON type. Provides context for what each end of the scale represents, enhancing evaluator understanding.

Parameter	Description	Required	Notes
`title`	The title of the anchor label	No	Provides a clear and concise title for the anchor label
`label_text`	The text displayed for the anchor label. `left` and `right` should be provided for `COMPARISON` type.	Yes	This is the text shown to the evaluator

For COMPARISON type:

left and right should be provided for label_text.
title is optional.

{
  "anchor_label": {
    "title": "Clarity",
    "label_text": {
      "left": "Much worse",
      "right": "Much better"
    }
  }
}

Instruction: Provides key guidance to evaluators on how to approach the evaluation.

Parameter	Description	Required	Notes
`type`	Type of instruction. Options: `DO`, `WARNING`, `DONT`, `EXAMPLE`	Yes	Guides evaluators on how to approach the evaluation
`instruction`	The main instruction text	Yes	Provides key guidance to evaluators
`description`	Additional details or context for the instruction	No	Optional for all instruction types
`reference_files`	A list of reference files to set a benchmark for evaluators	No	Up to 3 items. Each item must be a dictionary with `path` and `type`, where `type` is one of `audio`, `reference`, or `target`.

Additional Notes:

Instruction is optional.
description and reference_files are optional but can provide valuable context.
reference_files uses this shape: [{"path": "./sample.wav", "type": "audio"}].

Annotation: Collects free-form text feedback from evaluators. Defined in the annotations array.

Parameter	Description	Required	Notes
`type`	Must be `ANNOTATION`	Yes	Identifies this as an annotation question
`question`	The question text shown to evaluators	Yes	Cannot be empty
`related_model`	Which audio the annotation applies to	Conditional	`ALL` for single stimulus, `MODEL_A` or `MODEL_B` for double stimulus
`description`	Additional guidance for evaluators	No	Optional hint for what feedback to provide

Important Notes:

The annotations array is separate from the questions array.
For single stimulus evaluations, use related_model: "ALL" or omit it.
For double stimulus evaluations, related_model must be "MODEL_A" or "MODEL_B".
Cannot use annotations array together with use_annotation=True parameter.

Example:

{
  "questions": [...],
  "annotations": [
    {
      "type": "ANNOTATION",
      "question": "Describe any issues you noticed",
      "related_model": "ALL",
      "description": "e.g., noise, pronunciation errors"
    }
  ]
}

flash_eval()

Run automatic evaluation on an audio file and return a FlashEvalResult.

string

required

Path to the audio file to evaluate.

string

Language code for model routing. Currently available: en-us, es-es. When omitted, the default en-us model is used. Ignored when category="noise_quality".

string

Evaluation category. Defaults to naturalness when omitted. Currently available: naturalness, noise_quality.

Returns a FlashEvalResult with these fields:

Field	Description
`naturalness`	Naturalness score when `category="naturalness"`
`noise_quality`	Noise quality score when `category="noise_quality"`
`file`	The evaluated `File` object
`id`	Optional evaluation ID
`message`	Optional service message

python

result = client.flash_eval(file_path="path/to/audio.wav")
print(result.naturalness)

spanish_result = client.flash_eval(file_path="path/to/audio_es.wav", language="es-es")
noise_result = client.flash_eval(file_path="path/to/audio.wav", category="noise_quality")
print(noise_result.noise_quality)

get_evaluation_list()

Returns a JSON containing all your evaluations.

evaluations = client.get_evaluation_list()
print(evaluations)

The output JSON looks like:

[
  {
    "id": "<UUID>",
    "title": "How natural my synthetic voices are",
    "internal_name": null,
    "description": "Used latest internal model. Epoch 10, alpha 0.1",
    "batch_size": 1,
    "status": "ACTIVE",
    "created_time": "2024-06-25T01:40:43.429Z",
    "updated_time": "2024-06-26T13:21:34.801Z"
  }
]

get_eval_template_info()

Gets detailed information about an evaluation template by its ID.

string

The unique identifier of the evaluation template to retrieve information for.

Returns a Python dictionary containing detailed template information.

template_info = client.get_eval_template_info("abcdef")
print(template_info)

The returned dictionary has this shape:

{
  "id": "<UUID>",
  "code": "abcdef",
  "title": "Voice Quality Assessment",
  "description": "Template for evaluating voice naturalness and quality",
  "language": Language.ENGLISH_AMERICAN,
  "eval_type": "Single",
  "created_time": datetime.datetime(2024, 6, 25, 1, 40, 43, 429000, tzinfo=datetime.timezone.utc),
  "updated_time": datetime.datetime(2024, 6, 26, 13, 21, 34, 801000, tzinfo=datetime.timezone.utc)
}

Field	Description
`id`	Unique identifier (UUID) of the template
`code`	Template code used for identification
`title`	Display name of the template
`description`	Detailed description of the template’s purpose
`language`	`Language` enum value from the template, such as `Language.ENGLISH_AMERICAN` for `en-us`
`eval_type`	Type of evaluation: `Single`, `Double`, or `Triple`
`created_time`	`datetime.datetime` value for when the template was created
`updated_time`	`datetime.datetime` value for when the template was last modified

get_stats_json_by_id()

Returns a list of JSONs containing the statistics of each stimulus for the evaluation referenced by the id.

string

Evaluation id. See get_evaluation_list().

string

default:"question"

Group by criteria. Options are “question”, “script”, or “model”. Default is “question”. Note that “script” and “model” are only available for single-question evaluations.

evaluations = client.get_evaluation_list()
for eval in evaluations:
    stats = client.get_stats_json_by_id(eval['id'], group_by='question')
    print(stats)

Field	Description	SCORED	NON_SCORED
`frequency`	List of score counts: `{"score": number, "count": number}[]`	✓	-
`mean`	Average score	✓	-
`median`	Median score	✓	-
`std`	Standard deviation	✓	-
`sem`	Standard error of the mean	✓	-
`ci_95`	95% confidence interval	✓	-
`options`	Each option name as key with integer value	✓	✓
`OTHER`	The number of evaluators who selected “Other”	✓	✓

For NON_SCORED questions:

The integer value is the number of evaluators who selected the option.
All options are included in the response regardless of their value

Single
Double (including CSMOS)

You can get the statistics of each question by calling get_stats_json_by_id() with group_by set to question, script, or model.

{
  "question": string,
  "description": string,
  "order": int,
  "responses": [
    {
      "name": string,
      "model_tag": string,
      "tags": string[],
      "type": "A" | "B" | "REF",
      "script": string | null,
      "frequency": [
        {
          "score": number,
          "count": number
        }
      ],
      "mean": float | null, // null if the question is not SCORED
      "median": float | null, // null if the question is not SCORED
      "std": float | null, // null if the question is not SCORED
      "sem": float | null, // null if the question is not SCORED
      "ci_95": float | null, // null if the question is not SCORED
    }
  ]
}

{
  "script": string,
  "responses": [
    {
      "question": string,
      "description": string,
      "order": int,
      "responses": [
        {
          "name": string,
          "model_tag": string,
          "tags": string[],
          "type": "A" | "B" | "REF",
          "script": string | null,
          "frequency": [
            {
              "score": number,
              "count": number
            }
          ],
          "mean": float | null, // null if the question is not SCORED
          "median": float | null, // null if the question is not SCORED
          "std": float | null, // null if the question is not SCORED
          "sem": float | null, // null if the question is not SCORED
          "ci_95": float | null, // null if the question is not SCORED
        },
        ...
      ]
    },
    ...
  ]
}

{
  "model": string,
  "responses": [
    {
      "question": string,
      "description": string,
      "order": int,
      "responses": [
        {
          "name": string,
          "model_tag": string,
          "tags": string[],
          "type": "A" | "B" | "REF",
          "script": string | null,
          "frequency": [
            {
              "score": number,
              "count": number
            }
          ],
          "mean": float | null, // null if the question is not SCORED
          "median": float | null, // null if the question is not SCORED
          "std": float | null, // null if the question is not SCORED
          "sem": float | null, // null if the question is not SCORED
          "ci_95": float | null, // null if the question is not SCORED
        },
        ...
      ]
    },
    ...
  ]
}

The group_by is set to question by default. Other options (script, model) cannot be used.

{
  "question": string,
  "description": string,
  "order": int,
  "responses": [
    {
      "targets": [
        {
          "name": string,
          "model_tag": string,
          "tags": string[],
          "type": "A" | "B" | "REF",
          "script": string | null,
        }
      ],
      "frequency": [
        {
          "score": number,
          "count": number
        }
      ],
      "mean": float | null, // null if the question is not SCORED
      "median": float | null, // null if the question is not SCORED
      "std": float | null, // null if the question is not SCORED
      "sem": float | null, // null if the question is not SCORED
      "ci_95": float | null, // null if the question is not SCORED
    }
  ]
}

download_evaluation_files_by_evaluation_id()

Download all files associated with a specific evaluation, identified by its evaluation_id, from the Podonos evaluation service. It saves these files to a specified directory on the local file system and generates a metadata file describing the downloaded files. Return a string indicating the status of the download operation. This could be a success message or an error message if the download fails.

string

Evaluation id. See get_evaluation_list().

string

The directory path where the downloaded files will be saved. This should be a valid path on the local file system where the user has write permissions.

client.download_evaluation_files_by_evaluation_id(
  evaluation_id="12345",
  output_dir="./output",
)

{
  "files": [
    {
      "file_path": "./output/model1/4c318c0fd5a492d3e20cb8142ac5344b.wav",
      "original_name": "file1.wav",
      "model_tag": "model1",
      "tags": ["tag1", "tag2"]
    }
  ]
}

Field	Description
`file_path`	The path produced by joining `output_dir` with `{model_tag}/{hashed_file_name}`. It can be relative or absolute depending on the `output_dir` you pass.
`original_name`	The original name of the file before downloading.
`model_tag`	The model tag associated with the file, used for categorization.
`tags`	A list of tags associated with the file, providing additional context or categorization.

File Naming Convention:Each downloaded file is saved in the format {output_dir}/{model_tag}/{file_name}. This means that files are organized into subdirectories named after their model_tag, and the original file name is hashed formatted.

File

A class representing one file, used for adding files in Evaluator.

string

required

Path to the file to evaluate. For audio files, we support wav, mp3, and flac formats.

string

required

Name of your model (e.g., WhisperTTS) or any unique name (e.g., human).

list[string]

A list of string tags for the file designated by path. You can use this field as properties of the file such as original, synthesized, tom, maria, and so on. Later you can look up or group files with particular tags in the output report.

string

Text script of the input audio file.

bool

default:"False"

True if this file works as a reference in a comparative evaluation.

Dict[str, str | int | float | bool | None]

Optional metadata dictionary for the file. Keys must be strings, and values must be JSON-primitive values.

list[string]

Optional tags for the script. These are useful for ranking and script-level grouping.

python

file = File(
    path="/path/to/speech.wav",
    model_tag="my_model",
    tags=["synthesized", "male"],
    script="hello there",
    meta_data={"speaker_id": "spk-001"},
    script_tags=["greeting"],
)

Evaluator

Evaluator manages a single type of evaluation.

add_file()

Add one file to evaluate in a single evaluation question. For a single file evaluation like NMOS, one file to evaluate is added.

File

required

Input File. This field is required if type is NMOS, QMOS, P808, or CUSTOM_SINGLE.

etor.add_file(
    file=File(
        path="/path/to/speech_0_0.wav",
        model_tag="my_model",
        tags=["synthesized", "male", "ver1234"],
    )
)

etor.add_file(
    file=File(
        path="/path/to/speech_0_0.wav",
        model_tag="my_model",
        tags=["synthesized", "female", "ver1234"],
    )
)

add_files()

Add multiple files for evaluations that require comparison.

File

required

First input file.

File

required

Second input file.

File

Optional third input file. Used for CSMOS evaluations with one reference and two stimuli.

The order and reference requirements depend on evaluation type:

Type	File requirements
`PREF`, `CUSTOM_DOUBLE`	Two ordered stimulus files
`SMOS`	Two unordered stimulus files
`CMOS`	One reference file and one stimulus file
`CSMOS`	Two stimulus files followed by one reference file (`file2`)

file0 = File(path="/path/to/speech0.wav", model_tag="human", tags=["original", "male"])
file1 = File(path="/path/to/speech1.wav", model_tag="my_model", tags=["synthesized", "male", "ver1234"])
etor.add_files(file0=file0, file1=file1)

reference = File(path="/path/to/reference.wav", model_tag="human", is_ref=True)
model_a = File(path="/path/to/model_a.wav", model_tag="model_a")
model_b = File(path="/path/to/model_b.wav", model_tag="model_b")
etor.add_files(file0=model_a, file1=model_b, file2=reference)

add_ranking_set()

Add one ranking set for a RANKING evaluation.

list[File]

required

Ordered candidate files for one ranking group. Files must be stimuli, not references.

Constraints enforced across calls:

All groups must have the same number of files.
The order of model_tag must be identical across groups.
Files must be stimuli; do not set is_ref=True.

python

etor = client.create_evaluator(type="RANKING", name="Ranking test")

etor.add_ranking_set([
    File(path="/path/to/model_a_001.wav", model_tag="model_a", script_tags=["script_001"]),
    File(path="/path/to/model_b_001.wav", model_tag="model_b", script_tags=["script_001"]),
    File(path="/path/to/model_c_001.wav", model_tag="model_c", script_tags=["script_001"]),
])

close()

Close the evaluation session. Once this function is called, all the evaluation files will be sent to the Podonos evaluation service, the files will go through a series of processing, and delivered to evaluators. Returns a JSON object containing the uploading status.

python

status = etor.close()

Get Started

Basics

Details

Use Cases

Roadmap

SDK References

Podonos

init()

Client

create_evaluator()

resume_evaluator()

create_evaluator_from_template()

create_evaluator_from_template_json()

flash_eval()

get_evaluation_list()

get_eval_template_info()

get_stats_json_by_id()

download_evaluation_files_by_evaluation_id()

File

Evaluator

add_file()

add_files()

add_ranking_set()

close()

​Podonos

​init()

​Client

​create_evaluator()

​resume_evaluator()

​create_evaluator_from_template()

​create_evaluator_from_template_json()

​flash_eval()

​get_evaluation_list()

​get_eval_template_info()

​get_stats_json_by_id()

​download_evaluation_files_by_evaluation_id()

​File

​Evaluator

​add_file()

​add_files()

​add_ranking_set()

​close()

Podonos

init()

Client

create_evaluator()

resume_evaluator()

create_evaluator_from_template()

create_evaluator_from_template_json()

flash_eval()

get_evaluation_list()

get_eval_template_info()

get_stats_json_by_id()

download_evaluation_files_by_evaluation_id()

File

Evaluator

add_file()

add_files()

add_ranking_set()

close()