> ## Documentation Index
> Fetch the complete documentation index at: https://podonos.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Reference

> Details of modules and classes

## Podonos

This is a base module. You can import:

```python python theme={null}
import podonos
from podonos import *
```

### init()

Initialize the module and return an instance of `Client`.

<ParamField query="api_key" type="string">
  API key you obtained from the workspace. For details, see the [Get API key](/docs/apikey).
  If this is not set, the package tries to read `PODONOS_API_KEY` from the environment variable.
  Throws an error if both of them are not available.
</ParamField>

<ParamField query="api_url" default="https://prod.podonosapi.com" type="string">
  API base URL. You usually do not need to set this unless you are using a private, staging, or development API endpoint.
</ParamField>

Returns an instance of `Client`.

```python python theme={null}
client = podonos.init(api_key="<API_KEY>")

# Optional: pass the production API endpoint explicitly
client = podonos.init(api_key="<API_KEY>", api_url="https://prod.podonosapi.com")
```

## Client

`Client` manages one or more `Evaluator` instances and evaluation history.

### create\_evaluator()

Create a new instance of `Evaluator`. One evaluator supports a single type of evaluation throughout its life cycle.
If you want multiple types of evaluation, create multiple evaluators by calling `create_evaluator()` multiple times.

<ParamField query="name" type="string">
  Name of this evaluation session. If empty, a random name is automatically generated and used.
</ParamField>

<ParamField query="desc" type="string">
  Description of this evaluation session. This field is for your records, so later you can see how you generated the output files or trained your model.
</ParamField>

<ParamField query="type" default="NMOS" type="string">
  Evaluation type. One of the following:

  | Type            | Description                                                             |
  | --------------- | ----------------------------------------------------------------------- |
  | `NMOS`          | Naturalness Mean Opinion Score                                          |
  | `QMOS`          | Quality Mean Opinion Score                                              |
  | `CMOS`          | Reference-based comparison (compare target audio against reference)     |
  | `CSMOS`         | Comparative similarity test between two target audios and one reference |
  | `SMOS`          | Similarity Mean Opinion Score                                           |
  | `P808`          | Speech quality by ITU-T P.808                                           |
  | `PREF`          | Preference test between two audios/speeches                             |
  | `CUSTOM_SINGLE` | Custom single-stimulus evaluation                                       |
  | `CUSTOM_DOUBLE` | Custom double-stimulus evaluation                                       |
  | `RANKING`       | Ranking evaluation across multiple stimuli                              |
</ParamField>

<ParamField query="lan" default="en-us" type="string">
  Specific language and locale of the speech. Currently we support:

  | Code    | Description              |
  | ------- | ------------------------ |
  | `en-us` | English (United States)  |
  | `en-gb` | English (United Kingdom) |
  | `en-au` | English (Australia)      |
  | `en-ca` | English (Canada)         |
  | `en-in` | English (India)          |
  | `ko-kr` | Korean (Korea)           |
  | `zh-cn` | Mandarin (China)         |
  | `es-es` | Spanish (Spain)          |
  | `es-mx` | Spanish (Mexico)         |
  | `fr-fr` | French (France)          |
  | `fr-ca` | French (Canada)          |
  | `de-de` | German (Germany)         |
  | `ja-jp` | Japanese (Japan)         |
  | `it-it` | Italian (Italy)          |
  | `pl-pl` | Polish (Poland)          |
  | `pt-pt` | Portuguese (Portugal)    |
  | `pt-br` | Portuguese (Brazil)      |
  | `id-id` | Indonesian (Indonesia)   |
  | `nl-nl` | Dutch (Netherlands)      |
  | `sv-se` | Swedish (Sweden)         |
  | `hi-in` | Hindi (India)            |
  | `ta-in` | Tamil (India)            |
  | `kn-in` | Kannada (India)          |
  | `ml-in` | Malayalam (India)        |
  | `si-lk` | Sinhala (Sri Lanka)      |
  | `ar-eg` | Arabic (Egypt)           |
  | `ar-ae` | Arabic (UAE)             |
  | `ar-sa` | Arabic (Saudi Arabia)    |
  | `audio` | General audio file       |

  We will add more soon. Please check later again.
</ParamField>

<ParamField query="granularity" default="1.0" type="float">
  Granularity of evaluation scales. Supported values are `1.0` and `0.5`.
</ParamField>

<ParamField query="num_eval" default="10" type="int">
  Number of evaluations per sample. For example, if this is 10 for NMOS type evaluation, each audio file will be assigned to 10 humans, and the statistics of the evaluation output will be computed and presented in the final report.
</ParamField>

<ParamField query="due_hours" default="12" type="int">
  Expected due time of the final report in hours. Must be at least `12`. Depending on the hours, the pricing may change.
</ParamField>

<ParamField query="use_annotation" default="False" type="bool">
  Enable annotation to collect free-form text feedback from evaluators.
</ParamField>

<ParamField query="use_loudness_normalization" default="True" type="bool">
  Enable loudness normalization to ensure consistent audio volume levels during evaluation.
</ParamField>

<ParamField query="auto_start" default="False" type="bool">
  If True, the evaluation automatically starts when you finish uploading the files. If False, you go to [Workspace](https://workspace.podonos.com), confirm the evaluation session, and manually start the evaluation.
</ParamField>

<ParamField query="max_upload_workers" default="20" type="int">
  Maximum number of upload worker threads. If you experience a slow upload, please increase the number of workers.
</ParamField>

<ParamField query="verify_batch_size" default="100" type="int">
  Batch size for file verification API calls. Must be between `1` and `1000`.
</ParamField>

<ParamField query="api_timeout" default="(5, 30)" type="tuple[float, float]">
  Timeout tuple for API requests, in seconds.
</ParamField>

<ParamField query="verify_timeout" default="(5, 120)" type="tuple[float, float]">
  Timeout tuple for file verification requests, in seconds.
</ParamField>

<ParamField query="upload_timeout" default="(10, 300)" type="tuple[float, float]">
  Timeout tuple for direct file upload requests, in seconds.
</ParamField>

<ParamField query="resume_upload" default="False" type="bool">
  Enable SDK-local upload ledger/resume support. The ledger records upload progress and the original evaluation contract so interrupted uploads can be resumed safely.
</ParamField>

<ParamField query="upload_state_path" default="None" type="string">
  Optional SQLite upload ledger path when `resume_upload=True`. Use the same path with `resume_evaluator()` if the upload is interrupted.
</ParamField>

Returns an instance of `Evaluator`.

<CodeGroup>
  ```python simplest theme={null}
  etor = client.create_evaluator()
  ```

  ```python name and description theme={null}
  etor = client.create_evaluator(
      name="sim_iter100",
      desc="new_model_vs_sota",
      type="SMOS",
  )
  ```

  ```python full theme={null}
  etor = client.create_evaluator(
      name="sim_iter100",
      desc="new_model_vs_sota",
      type="SMOS",
      lan="en-us",
      granularity=1.0,
      num_eval=100,
      due_hours=24,
      use_loudness_normalization=True,
      auto_start=True,
      verify_batch_size=100,
      resume_upload=True,
      upload_state_path="./podonos-upload-ledger.sqlite3",
  )
  ```
</CodeGroup>

### resume\_evaluator()

Resume uploads for an existing evaluation using an SDK upload ledger. Add files again in the same order as the interrupted run so the ledger can recover each file's remote object identity.

<Note>
  Resume is safety-checked against the local ledger. The ledger must contain the original evaluation contract and session configuration for `evaluation_id`; otherwise the SDK refuses to resume. The SDK restores the original session details from the ledger where available, so do not use `resume_evaluator()` to change the evaluation type, language, template, batch size, or file identities.

  During resume, completed ledger rows are reused only when the current local file still matches the recorded path, content hash, size, and upload manifest. If a completed metadata or verification row no longer matches the local file, start a fresh evaluation or remove the stale ledger row before resuming.
</Note>

<ParamField query="evaluation_id" type="string" required>
  Evaluation ID to resume.
</ParamField>

<ParamField query="upload_state_path" type="string" required>
  Path to the SDK upload ledger created by a previous upload run. It must contain the original evaluation contract for `evaluation_id`.
</ParamField>

<ParamField query="name" type="string">
  Optional session name.
</ParamField>

<ParamField query="desc" type="string">
  Optional session description.
</ParamField>

<ParamField query="type" default="NMOS" type="string">
  Evaluation type. Uses the same supported values as `create_evaluator()`.
</ParamField>

<ParamField query="lan" default="en-us" type="string">
  Language code. Uses the same supported values as `create_evaluator()`.
</ParamField>

<ParamField query="granularity" default="1.0" type="float">
  Granularity of evaluation scales.
</ParamField>

<ParamField query="num_eval" default="10" type="int">
  Number of evaluations per sample.
</ParamField>

<ParamField query="due_hours" default="12" type="int">
  Expected due time of the final report in hours.
</ParamField>

<ParamField query="use_annotation" default="False" type="bool">
  Enable annotation to collect free-form text feedback from evaluators.
</ParamField>

<ParamField query="use_loudness_normalization" default="True" type="bool">
  Enable loudness normalization for evaluation.
</ParamField>

<ParamField query="auto_start" default="False" type="bool">
  If True, the evaluation automatically starts after files are uploaded.
</ParamField>

<ParamField query="max_upload_workers" default="20" type="int">
  Maximum number of upload worker threads.
</ParamField>

<ParamField query="verify_batch_size" default="100" type="int">
  Batch size for file verification API calls.
</ParamField>

<ParamField query="api_timeout" default="(5, 30)" type="tuple[float, float]">
  Timeout tuple for API requests, in seconds.
</ParamField>

<ParamField query="verify_timeout" default="(5, 120)" type="tuple[float, float]">
  Timeout tuple for file verification requests, in seconds.
</ParamField>

<ParamField query="upload_timeout" default="(10, 300)" type="tuple[float, float]">
  Timeout tuple for direct file upload requests, in seconds.
</ParamField>

```python python theme={null}
etor = client.resume_evaluator(
    evaluation_id="<EVALUATION_ID>",
    upload_state_path="./podonos-upload-ledger.sqlite3",
)
```

### create\_evaluator\_from\_template()

When you create an evaluation using a template, all the questions and options defined in the template are automatically assigned to the new evaluation. This ensures consistency and saves time by reusing pre-defined content.

<ParamField query="name" type="string" required>
  Name of this evaluation session. Required and must be non-empty.
</ParamField>

<ParamField query="template_id" type="string" required>
  The unique identifier of the template to base the new evaluation on.
</ParamField>

<ParamField query="desc" type="string">
  Description of this evaluation session. This field is for your records.
</ParamField>

<ParamField query="num_eval" default="10" type="int">
  Number of evaluations per sample.
</ParamField>

<ParamField query="use_annotation" default="False" type="bool">
  Enable annotation to collect free-form text feedback from evaluators. Cannot be used together with an `annotations` array in template JSON.
</ParamField>

<ParamField query="use_loudness_normalization" default="True" type="bool">
  Enable loudness normalization to ensure consistent audio volume levels during evaluation.
</ParamField>

<ParamField query="auto_start" default="False" type="bool">
  If True, the evaluation automatically starts when you finish uploading the files.
</ParamField>

<ParamField query="max_upload_workers" default="20" type="int">
  Maximum number of upload worker threads.
</ParamField>

<ParamField query="verify_batch_size" default="100" type="int">
  Batch size for file verification API calls.
</ParamField>

<ParamField query="api_timeout" default="(5, 30)" type="tuple[float, float]">
  Timeout tuple for API requests, in seconds.
</ParamField>

<ParamField query="verify_timeout" default="(5, 120)" type="tuple[float, float]">
  Timeout tuple for file verification requests, in seconds.
</ParamField>

<ParamField query="upload_timeout" default="(10, 300)" type="tuple[float, float]">
  Timeout tuple for direct file upload requests, in seconds.
</ParamField>

<ParamField query="resume_upload" default="False" type="bool">
  Enable SDK-local upload ledger/resume support. The ledger records upload progress and the original evaluation contract so interrupted uploads can be resumed safely.
</ParamField>

<ParamField query="upload_state_path" default="None" type="string">
  Optional SQLite upload ledger path when `resume_upload=True`. Use the same path with `resume_evaluator()` if the upload is interrupted.
</ParamField>

<CodeGroup>
  ```python python theme={null}
  etor = client.create_evaluator_from_template(
      name="Voice naturalness evaluation",
      desc="new_model_vs_competitor_model",
      num_eval=10,
      template_id="abcdef",
      use_loudness_normalization=True,
  )
  ```
</CodeGroup>

### create\_evaluator\_from\_template\_json()

Create a new evaluation using a JSON template. This allows you to define custom evaluation structures programmatically.

<ParamField query="json" type="Dict">
  Template JSON as a dictionary. Optional if `json_file` is provided.
</ParamField>

<ParamField query="json_file" type="string">
  Path to the JSON template file. Optional if `json` is provided.
</ParamField>

<ParamField query="name" type="string">
  Name of this evaluation session. Optional; if omitted, the SDK generates a name.
</ParamField>

<ParamField query="custom_type" type="string | CustomType" default="SINGLE">
  Type of evaluation. Accepts either a string or `CustomType` enum value.

  | Value        | Description                             | batch\_size | File Configuration       |
  | ------------ | --------------------------------------- | ----------- | ------------------------ |
  | `SINGLE`     | Single stimulus evaluation              | 1           | 1 stimulus               |
  | `DOUBLE`     | Double stimulus evaluation              | 2           | 2 stimuli (no reference) |
  | `SINGLE_REF` | Reference-based comparison (CMOS style) | 2           | 1 reference + 1 stimulus |
  | `RANKING`    | Ranking evaluation                      | 2+          | Multiple stimuli         |

  You can use string values (`"SINGLE"`, `"DOUBLE"`, `"SINGLE_REF"`, `"RANKING"`) or the `CustomType` enum from `podonos.common.enum`:

  ```python theme={null}
  from podonos.common.enum import CustomType

  custom_type=CustomType.SINGLE_REF
  custom_type="SINGLE_REF"
  ```
</ParamField>

<ParamField query="desc" type="string">
  Description of this evaluation session. Optional.
</ParamField>

<ParamField query="lan" default="en-us" type="string">
  Language for evaluation. See supported languages in `create_evaluator()`.
</ParamField>

<ParamField query="num_eval" default="10" type="int">
  Number of evaluations per sample.
</ParamField>

<ParamField query="use_annotation" default="False" type="bool">
  Enable annotation to collect free-form text feedback from evaluators. Cannot be used together with an `annotations` array in template JSON. When using custom annotation questions, define them in the `annotations` array instead.
</ParamField>

<ParamField query="use_loudness_normalization" default="True" type="bool">
  Enable loudness normalization to ensure consistent audio volume levels during evaluation.
</ParamField>

<ParamField query="auto_start" default="False" type="bool">
  If True, the evaluation automatically starts when you finish uploading the files.
</ParamField>

<ParamField query="max_upload_workers" default="20" type="int">
  Maximum number of upload workers. Must be a positive integer.
</ParamField>

<ParamField query="verify_batch_size" default="100" type="int">
  Batch size for file verification API calls.
</ParamField>

<ParamField query="api_timeout" default="(5, 30)" type="tuple[float, float]">
  Timeout tuple for API requests, in seconds.
</ParamField>

<ParamField query="verify_timeout" default="(5, 120)" type="tuple[float, float]">
  Timeout tuple for file verification requests, in seconds.
</ParamField>

<ParamField query="upload_timeout" default="(10, 300)" type="tuple[float, float]">
  Timeout tuple for direct file upload requests, in seconds.
</ParamField>

<ParamField query="resume_upload" default="False" type="bool">
  Enable SDK-local upload ledger/resume support. The ledger records upload progress and the original evaluation contract so interrupted uploads can be resumed safely.
</ParamField>

<ParamField query="upload_state_path" default="None" type="string">
  Optional SQLite upload ledger path when `resume_upload=True`. Use the same path with `resume_evaluator()` if the upload is interrupted.
</ParamField>

<CodeGroup>
  ```python Dictionary theme={null}
  # Using JSON dictionary
  template = {
      "questions": [
          {
              "type": "SCORED",
              "question": "How natural is the voice?",
              "description": "Rate the quality of the voice",
              "options": [
                  {"label_text": "Excellent"},
                  {"label_text": "Good"},
                  {"label_text": "Fair"},
                  {"label_text": "Poor"},
                  {"label_text": "Bad"},
              ],
          }
      ]
  }

  evaluator = client.create_evaluator_from_template_json(
      json=template,
      name="Quality Test",
      custom_type="SINGLE",
  )
  ```

  ```python JSON File theme={null}
  # Using JSON file
  evaluator = client.create_evaluator_from_template_json(
      json_file="./template.json",
      name="Quality Test",
      custom_type="DOUBLE",
  )
  ```

  ```python SINGLE_REF (CMOS style) theme={null}
  from podonos import File
  from podonos.common.enum import CustomType

  # CMOS-style reference comparison
  template = {
      "questions": [
          {
              "type": "SCORED",
              "question": "How similar is the synthesized audio to the reference?",
              "options": [
                  {"label_text": "Identical"},
                  {"label_text": "Very similar"},
                  {"label_text": "Somewhat similar"},
                  {"label_text": "Different"},
                  {"label_text": "Completely different"},
              ],
          }
      ]
  }

  evaluator = client.create_evaluator_from_template_json(
      json=template,
      name="CMOS Style Evaluation",
      custom_type=CustomType.SINGLE_REF,  # or "SINGLE_REF"
  )

  # Add files: 1 stimulus + 1 reference
  evaluator.add_files(
      file0=File(path="synthesized.wav", model_tag="tts"),
      file1=File(path="reference.wav", model_tag="human", is_ref=True),
  )
  ```
</CodeGroup>

Returns an instance of `Evaluator`.

Here's the JSON template for reference:

<Tabs>
  <Tab title="Question">
    `Question`: Represents the main question posed to evaluators about the audio being assessed. It guides evaluators on the specific aspect of the audio they should focus on during the evaluation.

    | Parameter        | Description                                                              | Required             | Notes                                                                 |
    | ---------------- | ------------------------------------------------------------------------ | -------------------- | --------------------------------------------------------------------- |
    | `type`           | Type of question. Options: `SCORED`, `NON_SCORED`, `COMPARISON`          | Yes                  | Determines the structure and requirements of the question             |
    | `question`       | The main question text                                                   | Yes                  | Must be provided for all question types                               |
    | `description`    | Additional details or context for the question                           | No                   | Optional for all question types                                       |
    | `options`        | List of possible options. Only for `SCORED` and `NON_SCORED` types       | Conditional          | Must have between 1 and 9 options for `SCORED` and `NON_SCORED` types |
    | `scale`          | Scale for comparison. Only for `COMPARISON` type                         | Conditional          | Must be an integer between 2 and 9 for `COMPARISON` type              |
    | `allow_multiple` | Allows multiple selections. Only for `NON_SCORED` type                   | Yes for `NON_SCORED` | Enables multiple choice selection                                     |
    | `has_other`      | Includes an "Other" option. Only for `NON_SCORED` type                   | No                   | Adds an option for evaluators to specify an unlisted choice           |
    | `has_none`       | Includes a "None" option. Only for `NON_SCORED` type                     | No                   | Adds an option for evaluators to select none of the listed choices    |
    | `related_model`  | Related model for the question. Only for `Double` Evaluation type.       | Conditional          | Select which model the question is related to.                        |
    | `anchor_label`   | Labels for the ends of the comparison scale. Only for `COMPARISON` type. | Conditional          | Provides context for what each end of the scale represents.           |

    <Note>
      Important Notes:

      * `SCORED` and `NON_SCORED` questions can have a maximum of 9 options.
      * `NON_SCORED` questions must specify `allow_multiple`.
      * `COMPARISON` type questions must have a scale between 2 and 9.
      * `related_model` consists of `ALL`, `MODEL_A` and `MODEL_B`. Default is `ALL`. The `related_model` is only used for the question (not for instructions).
    </Note>
  </Tab>

  <Tab title="Option">
    `Option`: Represents a possible answer or choice in a question. It provides evaluators with a range of options to choose from.

    | Parameter        | Description                                                                                       | Required | Notes                                                                  |
    | ---------------- | ------------------------------------------------------------------------------------------------- | -------- | ---------------------------------------------------------------------- |
    | `label_text`     | The text displayed for the option                                                                 | Yes      | This is the text shown to the evaluator                                |
    | `reference_file` | A reference audio file for the option. It helps the evaluator understand the quality of the audio | No       | This file serves as a benchmark and is saved in the evaluation results |

    <Note>
      For `options` in `SCORED` and `NON_SCORED` questions:

      * `score` is automatically generated only for `SCORED` questions. If there are 5 options, the first option in the list receives a score of 5, the second option receives a score of 4, and so on, down to a score of 1 for the last option.
      * `order` is the index of the option in the list, starting from 0.
    </Note>
  </Tab>

  <Tab title="Anchor Label">
    `Anchor Label`: Only for `COMPARISON` type. Provides context for what each end of the scale represents, enhancing evaluator understanding.

    | Parameter    | Description                                                                                           | Required | Notes                                                   |
    | ------------ | ----------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------- |
    | `title`      | The title of the anchor label                                                                         | No       | Provides a clear and concise title for the anchor label |
    | `label_text` | The text displayed for the anchor label. `left` and `right` should be provided for `COMPARISON` type. | Yes      | This is the text shown to the evaluator                 |

    <Note>
      For `COMPARISON` type:

      * `left` and `right` should be provided for `label_text`.
      * `title` is optional.

      ```json theme={null}
      {
        "anchor_label": {
          "title": "Clarity",
          "label_text": {
            "left": "Much worse",
            "right": "Much better"
          }
        }
      }
      ```
    </Note>
  </Tab>

  <Tab title="Instruction">
    `Instruction`: Provides key guidance to evaluators on how to approach the evaluation.

    | Parameter         | Description                                                      | Required | Notes                                                                                                                           |
    | ----------------- | ---------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------- |
    | `type`            | Type of instruction. Options: `DO`, `WARNING`, `DONT`, `EXAMPLE` | Yes      | Guides evaluators on how to approach the evaluation                                                                             |
    | `instruction`     | The main instruction text                                        | Yes      | Provides key guidance to evaluators                                                                                             |
    | `description`     | Additional details or context for the instruction                | No       | Optional for all instruction types                                                                                              |
    | `reference_files` | A list of reference files to set a benchmark for evaluators      | No       | Up to 3 items. Each item must be a dictionary with `path` and `type`, where `type` is one of `audio`, `reference`, or `target`. |

    <Note>
      Additional Notes:

      * `Instruction` is optional.
      * `description` and `reference_files` are optional but can provide valuable context.
      * `reference_files` uses this shape: `[{"path": "./sample.wav", "type": "audio"}]`.
    </Note>
  </Tab>

  <Tab title="Annotation">
    `Annotation`: Collects free-form text feedback from evaluators. Defined in the `annotations` array.

    | Parameter       | Description                           | Required    | Notes                                                                 |
    | --------------- | ------------------------------------- | ----------- | --------------------------------------------------------------------- |
    | `type`          | Must be `ANNOTATION`                  | Yes         | Identifies this as an annotation question                             |
    | `question`      | The question text shown to evaluators | Yes         | Cannot be empty                                                       |
    | `related_model` | Which audio the annotation applies to | Conditional | `ALL` for single stimulus, `MODEL_A` or `MODEL_B` for double stimulus |
    | `description`   | Additional guidance for evaluators    | No          | Optional hint for what feedback to provide                            |

    <Note>
      Important Notes:

      * The `annotations` array is separate from the `questions` array.
      * For single stimulus evaluations, use `related_model: "ALL"` or omit it.
      * For double stimulus evaluations, `related_model` must be `"MODEL_A"` or `"MODEL_B"`.
      * Cannot use `annotations` array together with `use_annotation=True` parameter.
    </Note>

    **Example:**

    ```json theme={null}
    {
      "questions": [...],
      "annotations": [
        {
          "type": "ANNOTATION",
          "question": "Describe any issues you noticed",
          "related_model": "ALL",
          "description": "e.g., noise, pronunciation errors"
        }
      ]
    }
    ```
  </Tab>
</Tabs>

### flash\_eval()

Run automatic evaluation on an audio file and return a `FlashEvalResult`.

<ParamField query="file_path" type="string" required>
  Path to the audio file to evaluate.
</ParamField>

<ParamField query="language" type="string">
  Language code for model routing. Currently available: `en-us`, `es-es`. When omitted, the default `en-us` model is used. Ignored when `category="noise_quality"`.
</ParamField>

<ParamField query="category" type="string">
  Evaluation category. Defaults to `naturalness` when omitted. Currently available: `naturalness`, `noise_quality`.
</ParamField>

Returns a `FlashEvalResult` with these fields:

| Field           | Description                                         |
| --------------- | --------------------------------------------------- |
| `naturalness`   | Naturalness score when `category="naturalness"`     |
| `noise_quality` | Noise quality score when `category="noise_quality"` |
| `file`          | The evaluated `File` object                         |
| `id`            | Optional evaluation ID                              |
| `message`       | Optional service message                            |

```python python theme={null}
result = client.flash_eval(file_path="path/to/audio.wav")
print(result.naturalness)

spanish_result = client.flash_eval(file_path="path/to/audio_es.wav", language="es-es")
noise_result = client.flash_eval(file_path="path/to/audio.wav", category="noise_quality")
print(noise_result.noise_quality)
```

### get\_evaluation\_list()

Returns a JSON containing all your evaluations.

<CodeGroup>
  ```python python theme={null}
  evaluations = client.get_evaluation_list()
  print(evaluations)
  ```
</CodeGroup>

The output JSON looks like:

```json theme={null}
[
  {
    "id": "<UUID>",
    "title": "How natural my synthetic voices are",
    "internal_name": null,
    "description": "Used latest internal model. Epoch 10, alpha 0.1",
    "batch_size": 1,
    "status": "ACTIVE",
    "created_time": "2024-06-25T01:40:43.429Z",
    "updated_time": "2024-06-26T13:21:34.801Z"
  }
]
```

### get\_eval\_template\_info()

Gets detailed information about an evaluation template by its ID.

<ParamField query="template_id" type="string">
  The unique identifier of the evaluation template to retrieve information for.
</ParamField>

Returns a Python dictionary containing detailed template information.

<CodeGroup>
  ```python python theme={null}
  template_info = client.get_eval_template_info("abcdef")
  print(template_info)
  ```
</CodeGroup>

The returned dictionary has this shape:

```python theme={null}
{
  "id": "<UUID>",
  "code": "abcdef",
  "title": "Voice Quality Assessment",
  "description": "Template for evaluating voice naturalness and quality",
  "language": Language.ENGLISH_AMERICAN,
  "eval_type": "Single",
  "created_time": datetime.datetime(2024, 6, 25, 1, 40, 43, 429000, tzinfo=datetime.timezone.utc),
  "updated_time": datetime.datetime(2024, 6, 26, 13, 21, 34, 801000, tzinfo=datetime.timezone.utc)
}
```

| Field          | Description                                                                              |
| -------------- | ---------------------------------------------------------------------------------------- |
| `id`           | Unique identifier (UUID) of the template                                                 |
| `code`         | Template code used for identification                                                    |
| `title`        | Display name of the template                                                             |
| `description`  | Detailed description of the template's purpose                                           |
| `language`     | `Language` enum value from the template, such as `Language.ENGLISH_AMERICAN` for `en-us` |
| `eval_type`    | Type of evaluation: `Single`, `Double`, or `Triple`                                      |
| `created_time` | `datetime.datetime` value for when the template was created                              |
| `updated_time` | `datetime.datetime` value for when the template was last modified                        |

### get\_stats\_json\_by\_id()

Returns a list of JSONs containing the statistics of each stimulus for the evaluation referenced by the `id`.

<ParamField query="evaluation_id" type="string">
  Evaluation id. See `get_evaluation_list()`.
</ParamField>

<ParamField query="group_by" default="question" type="string">
  Group by criteria. Options are "question", "script", or "model". Default is "question". Note that "script" and "model" are only available for single-question evaluations.
</ParamField>

<CodeGroup>
  ```python python theme={null}
  evaluations = client.get_evaluation_list()
  for eval in evaluations:
      stats = client.get_stats_json_by_id(eval['id'], group_by='question')
      print(stats)
  ```
</CodeGroup>

| Field       | Description                                                  | SCORED | NON\_SCORED |
| ----------- | ------------------------------------------------------------ | ------ | ----------- |
| `frequency` | List of score counts: `{"score": number, "count": number}[]` | ✓      | -           |
| `mean`      | Average score                                                | ✓      | -           |
| `median`    | Median score                                                 | ✓      | -           |
| `std`       | Standard deviation                                           | ✓      | -           |
| `sem`       | Standard error of the mean                                   | ✓      | -           |
| `ci_95`     | 95% confidence interval                                      | ✓      | -           |
| `options`   | Each option name as key with integer value                   | ✓      | ✓           |
| `OTHER`     | The number of evaluators who selected "Other"                | ✓      | ✓           |

<Note>
  For NON\_SCORED questions:

  * The integer value is the number of evaluators who selected the option.
  * All options are included in the response regardless of their value
</Note>

<Tabs>
  <Tab title="Single">
    You can get the statistics of each question by calling `get_stats_json_by_id()` with `group_by` set to `question`, `script`, or `model`.

    <CodeGroup>
      ```json question theme={null}
      {
        "question": string,
        "description": string,
        "order": int,
        "responses": [
          {
            "name": string,
            "model_tag": string,
            "tags": string[],
            "type": "A" | "B" | "REF",
            "script": string | null,
            "frequency": [
              {
                "score": number,
                "count": number
              }
            ],
            "mean": float | null, // null if the question is not SCORED
            "median": float | null, // null if the question is not SCORED
            "std": float | null, // null if the question is not SCORED
            "sem": float | null, // null if the question is not SCORED
            "ci_95": float | null, // null if the question is not SCORED
          }
        ]
      }
      ```

      ```json script theme={null}
      {
        "script": string,
        "responses": [
          {
            "question": string,
            "description": string,
            "order": int,
            "responses": [
              {
                "name": string,
                "model_tag": string,
                "tags": string[],
                "type": "A" | "B" | "REF",
                "script": string | null,
                "frequency": [
                  {
                    "score": number,
                    "count": number
                  }
                ],
                "mean": float | null, // null if the question is not SCORED
                "median": float | null, // null if the question is not SCORED
                "std": float | null, // null if the question is not SCORED
                "sem": float | null, // null if the question is not SCORED
                "ci_95": float | null, // null if the question is not SCORED
              },
              ...
            ]
          },
          ...
        ]
      }
      ```

      ```json model theme={null}
      {
        "model": string,
        "responses": [
          {
            "question": string,
            "description": string,
            "order": int,
            "responses": [
              {
                "name": string,
                "model_tag": string,
                "tags": string[],
                "type": "A" | "B" | "REF",
                "script": string | null,
                "frequency": [
                  {
                    "score": number,
                    "count": number
                  }
                ],
                "mean": float | null, // null if the question is not SCORED
                "median": float | null, // null if the question is not SCORED
                "std": float | null, // null if the question is not SCORED
                "sem": float | null, // null if the question is not SCORED
                "ci_95": float | null, // null if the question is not SCORED
              },
              ...
            ]
          },
          ...
        ]
      }
      ```
    </CodeGroup>
  </Tab>

  <Tab title="Double (including CSMOS)">
    The `group_by` is set to `question` by default. Other options (`script`, `model`) cannot be used.

    <CodeGroup>
      ```json question theme={null}
      {
        "question": string,
        "description": string,
        "order": int,
        "responses": [
          {
            "targets": [
              {
                "name": string,
                "model_tag": string,
                "tags": string[],
                "type": "A" | "B" | "REF",
                "script": string | null,
              }
            ],
            "frequency": [
              {
                "score": number,
                "count": number
              }
            ],
            "mean": float | null, // null if the question is not SCORED
            "median": float | null, // null if the question is not SCORED
            "std": float | null, // null if the question is not SCORED
            "sem": float | null, // null if the question is not SCORED
            "ci_95": float | null, // null if the question is not SCORED
          }
        ]
      }
      ```
    </CodeGroup>
  </Tab>
</Tabs>

### download\_evaluation\_files\_by\_evaluation\_id()

Download all files associated with a specific evaluation, identified by its `evaluation_id`, from the Podonos evaluation service. It saves these files to a specified directory on the local file system and generates a metadata file describing the downloaded files.
Return a string indicating the status of the download operation. This could be a success message or an error message if the download fails.

<ParamField query="evaluation_id" type="string">
  Evaluation id. See `get_evaluation_list()`.
</ParamField>

<ParamField query="output_dir" type="string">
  The directory path where the downloaded files will be saved. This should be a valid path on the local file system where the user has write permissions.
</ParamField>

<CodeGroup>
  ```python example theme={null}
  client.download_evaluation_files_by_evaluation_id(
    evaluation_id="12345",
    output_dir="./output",
  )
  ```

  ```json metadata.json theme={null}
  {
    "files": [
      {
        "file_path": "./output/model1/4c318c0fd5a492d3e20cb8142ac5344b.wav",
        "original_name": "file1.wav",
        "model_tag": "model1",
        "tags": ["tag1", "tag2"]
      }
    ]
  }
  ```
</CodeGroup>

| Field           | Description                                                                                                                                             |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `file_path`     | The path produced by joining `output_dir` with `{model_tag}/{hashed_file_name}`. It can be relative or absolute depending on the `output_dir` you pass. |
| `original_name` | The original name of the file before downloading.                                                                                                       |
| `model_tag`     | The model tag associated with the file, used for categorization.                                                                                        |
| `tags`          | A list of tags associated with the file, providing additional context or categorization.                                                                |

<Note>
  File Naming Convention:

  Each downloaded file is saved in the format `{output_dir}/{model_tag}/{file_name}`. This means that files are organized into subdirectories named after their `model_tag`, and the original file name is hashed formatted.
</Note>

## File

A class representing one file, used for adding files in `Evaluator`.

<ParamField query="path" type="string" required>
  Path to the file to evaluate. For audio files, we support `wav`, `mp3`, and `flac` formats.
</ParamField>

<ParamField query="model_tag" type="string" required>
  Name of your model (e.g., `WhisperTTS`) or any unique name (e.g., `human`).
</ParamField>

<ParamField query="tags" type="list[string]">
  A list of string tags for the file designated by `path`. You can use this field as properties of the file such as `original`, `synthesized`, `tom`, `maria`, and so on. Later you can look up or group files with particular tags in the output report.
</ParamField>

<ParamField query="script" type="string">
  Text script of the input audio file.
</ParamField>

<ParamField query="is_ref" default="False" type="bool">
  True if this file works as a reference in a comparative evaluation.
</ParamField>

<ParamField query="meta_data" type="Dict[str, str | int | float | bool | None]">
  Optional metadata dictionary for the file. Keys must be strings, and values must be JSON-primitive values.
</ParamField>

<ParamField query="script_tags" type="list[string]">
  Optional tags for the script. These are useful for ranking and script-level grouping.
</ParamField>

```python python theme={null}
file = File(
    path="/path/to/speech.wav",
    model_tag="my_model",
    tags=["synthesized", "male"],
    script="hello there",
    meta_data={"speaker_id": "spk-001"},
    script_tags=["greeting"],
)
```

## Evaluator

`Evaluator` manages a single type of evaluation.

### add\_file()

Add one file to evaluate in a single evaluation question. For a single file evaluation like `NMOS`, one file to evaluate is added.

<ParamField query="file" type="File" required>
  Input `File`. This field is required if `type` is `NMOS`, `QMOS`, `P808`, or `CUSTOM_SINGLE`.
</ParamField>

<CodeGroup>
  ```python NMOS theme={null}
  etor.add_file(
      file=File(
          path="/path/to/speech_0_0.wav",
          model_tag="my_model",
          tags=["synthesized", "male", "ver1234"],
      )
  )
  ```

  ```python P808 theme={null}
  etor.add_file(
      file=File(
          path="/path/to/speech_0_0.wav",
          model_tag="my_model",
          tags=["synthesized", "female", "ver1234"],
      )
  )
  ```
</CodeGroup>

### add\_files()

Add multiple files for evaluations that require comparison.

<ParamField query="file0" type="File" required>
  First input file.
</ParamField>

<ParamField query="file1" type="File" required>
  Second input file.
</ParamField>

<ParamField query="file2" type="File">
  Optional third input file. Used for `CSMOS` evaluations with one reference and two stimuli.
</ParamField>

The order and reference requirements depend on evaluation type:

| Type                    | File requirements                                           |
| ----------------------- | ----------------------------------------------------------- |
| `PREF`, `CUSTOM_DOUBLE` | Two ordered stimulus files                                  |
| `SMOS`                  | Two unordered stimulus files                                |
| `CMOS`                  | One reference file and one stimulus file                    |
| `CSMOS`                 | Two stimulus files followed by one reference file (`file2`) |

<CodeGroup>
  ```python SMOS theme={null}
  file0 = File(path="/path/to/speech0.wav", model_tag="human", tags=["original", "male"])
  file1 = File(path="/path/to/speech1.wav", model_tag="my_model", tags=["synthesized", "male", "ver1234"])
  etor.add_files(file0=file0, file1=file1)
  ```

  ```python CSMOS theme={null}
  reference = File(path="/path/to/reference.wav", model_tag="human", is_ref=True)
  model_a = File(path="/path/to/model_a.wav", model_tag="model_a")
  model_b = File(path="/path/to/model_b.wav", model_tag="model_b")
  etor.add_files(file0=model_a, file1=model_b, file2=reference)
  ```
</CodeGroup>

### add\_ranking\_set()

Add one ranking set for a `RANKING` evaluation.

<ParamField query="files" type="list[File]" required>
  Ordered candidate files for one ranking group. Files must be stimuli, not references.
</ParamField>

Constraints enforced across calls:

* All groups must have the same number of files.
* The order of `model_tag` must be identical across groups.
* Files must be stimuli; do not set `is_ref=True`.

```python python theme={null}
etor = client.create_evaluator(type="RANKING", name="Ranking test")

etor.add_ranking_set([
    File(path="/path/to/model_a_001.wav", model_tag="model_a", script_tags=["script_001"]),
    File(path="/path/to/model_b_001.wav", model_tag="model_b", script_tags=["script_001"]),
    File(path="/path/to/model_c_001.wav", model_tag="model_c", script_tags=["script_001"]),
])
```

### close()

Close the evaluation session. Once this function is called, all the evaluation files will be sent to the Podonos evaluation service, the files will go through a series of processing, and delivered to evaluators.

Returns a JSON object containing the uploading status.

```python python theme={null}
status = etor.close()
```