> ## Documentation Index
> Fetch the complete documentation index at: https://podonos.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Reference

> Details of modules and classes

## Podonos

This is a base module. You can import:

```python python theme={null}
import podonos
from podonos import *
```

### init()

Initialize the module and return an instance of `Client`.

<ParamField query="api_key" type="string">
  API key you obtained from the workspace. For details, see the [Get API key](/docs/apikey).
  If this is not set, the package tries to read `PODONOS_API_KEY` from the environment variable.
  Throws an error if both of them are not available.
</ParamField>

<ParamField query="api_url" default="https://prod.podonosapi.com" type="string">
  API base URL. You usually do not need to set this unless you are using a private, staging, or development API endpoint.
</ParamField>

Returns an instance of `Client`.

```python python theme={null}
client = podonos.init(api_key="<API_KEY>")

# Optional: pass the production API endpoint explicitly
client = podonos.init(api_key="<API_KEY>", api_url="https://prod.podonosapi.com")
```

## Client

`Client` manages one or more `Evaluator` instances and evaluation history.

### create\_evaluator()

Create a new instance of `Evaluator`. One evaluator supports a single type of evaluation throughout its life cycle.
If you want multiple types of evaluation, create multiple evaluators by calling `create_evaluator()` multiple times.

<ParamField query="name" type="string">
  Name of this evaluation session. If empty, a random name is automatically generated and used.
</ParamField>

<ParamField query="desc" type="string">
  Description of this evaluation session. This field is for your records, so later you can see how you generated the output files or trained your model.
</ParamField>

<ParamField query="type" default="NMOS" type="string">
  Evaluation type. One of the following:

  | Type            | Description                                                             |
  | --------------- | ----------------------------------------------------------------------- |
  | `NMOS`          | Naturalness Mean Opinion Score                                          |
  | `QMOS`          | Quality Mean Opinion Score                                              |
  | `CMOS`          | Reference-based comparison (compare target audio against reference)     |
  | `CSMOS`         | Comparative similarity test between two target audios and one reference |
  | `SMOS`          | Similarity Mean Opinion Score                                           |
  | `P808`          | Speech quality by ITU-T P.808                                           |
  | `PREF`          | Preference test between two audios/speeches                             |
  | `CUSTOM_SINGLE` | Custom single-stimulus evaluation                                       |
  | `CUSTOM_DOUBLE` | Custom double-stimulus evaluation                                       |
  | `RANKING`       | Ranking evaluation across multiple stimuli                              |
  | `RANKING_REF`   | Ranking across multiple stimuli against one reference per set           |
</ParamField>

<ParamField query="lan" default="en-us" type="string">
  Specific language and locale of the speech. Currently we support:

  | Code    | Description              |
  | ------- | ------------------------ |
  | `en-us` | English (United States)  |
  | `en-gb` | English (United Kingdom) |
  | `en-au` | English (Australia)      |
  | `en-ca` | English (Canada)         |
  | `en-in` | English (India)          |
  | `ko-kr` | Korean (Korea)           |
  | `zh-cn` | Mandarin (China)         |
  | `es-es` | Spanish (Spain)          |
  | `es-mx` | Spanish (Mexico)         |
  | `fr-fr` | French (France)          |
  | `fr-ca` | French (Canada)          |
  | `de-de` | German (Germany)         |
  | `ja-jp` | Japanese (Japan)         |
  | `it-it` | Italian (Italy)          |
  | `pl-pl` | Polish (Poland)          |
  | `pt-pt` | Portuguese (Portugal)    |
  | `pt-br` | Portuguese (Brazil)      |
  | `id-id` | Indonesian (Indonesia)   |
  | `nl-nl` | Dutch (Netherlands)      |
  | `sv-se` | Swedish (Sweden)         |
  | `hi-in` | Hindi (India)            |
  | `ta-in` | Tamil (India)            |
  | `kn-in` | Kannada (India)          |
  | `ml-in` | Malayalam (India)        |
  | `si-lk` | Sinhala (Sri Lanka)      |
  | `bn-in` | Bengali (India)          |
  | `gu-in` | Gujarati (India)         |
  | `mr-in` | Marathi (India)          |
  | `te-in` | Telugu (India)           |
  | `ar-eg` | Arabic (Egypt)           |
  | `ar-ae` | Arabic (UAE)             |
  | `ar-sa` | Arabic (Saudi Arabia)    |
  | `audio` | General audio file       |

  We will add more soon. Please check later again.
</ParamField>

<ParamField query="granularity" default="1.0" type="float">
  Granularity of evaluation scales. Supported values are `1.0` and `0.5`.
</ParamField>

<ParamField query="num_eval" default="10" type="int">
  Number of evaluations per sample. For example, if this is 10 for NMOS type evaluation, each audio file will be assigned to 10 humans, and the statistics of the evaluation output will be computed and presented in the final report.
</ParamField>

<ParamField query="due_hours" default="12" type="int">
  Expected due time of the final report in hours. Must be at least `12`. Depending on the hours, the pricing may change.
</ParamField>

<ParamField query="use_annotation" default="False" type="bool">
  Enable annotation to collect free-form text feedback from evaluators.
</ParamField>

<ParamField query="use_loudness_normalization" default="True" type="bool">
  Enable loudness normalization to ensure consistent audio volume levels during evaluation.
</ParamField>

<ParamField query="auto_start" default="False" type="bool">
  If `True`, `close()` waits for the uploaded files to finish processing, then starts the evaluation and **charges your workspace balance**. It raises if it cannot start. If `False`, `close()` returns as soon as the uploads are done and nothing is charged until you start the evaluation yourself in [Workspace](https://workspace.podonos.com). Requires podonos 0.46.0 or newer; earlier versions accept the parameter and silently ignore it.

  Pair it with `resume_upload=True` and an `upload_state_path`. If the start fails or times out, `resume_evaluator()` is the only way back to that evaluation; without a ledger you have to start it from Workspace instead.
</ParamField>

<ParamField query="max_upload_workers" default="20" type="int">
  Maximum number of upload worker threads. If you experience a slow upload, please increase the number of workers.
</ParamField>

<ParamField query="verify_batch_size" default="100" type="int">
  Batch size for file verification API calls. Must be between `1` and `1000`.
</ParamField>

<ParamField query="api_timeout" default="(5, 30)" type="tuple[float, float]">
  Timeout tuple for API requests, in seconds.
</ParamField>

<ParamField query="verify_timeout" default="(5, 120)" type="tuple[float, float]">
  Timeout tuple for file verification requests, in seconds.
</ParamField>

<ParamField query="upload_timeout" default="(10, 300)" type="tuple[float, float]">
  Timeout tuple for direct file upload requests, in seconds.
</ParamField>

<ParamField query="resume_upload" default="False" type="bool">
  Enable SDK-local upload ledger/resume support. The ledger records upload progress and the original evaluation contract so interrupted uploads can be resumed safely.
</ParamField>

<ParamField query="upload_state_path" default="None" type="string">
  Optional SQLite upload ledger path when `resume_upload=True`. Use the same path with `resume_evaluator()` if the upload is interrupted.
</ParamField>

<ParamField query="start_timeout" default="1800" type="float">
  Seconds `close()` waits when `auto_start=True`. Must be positive. It limits when the SDK stops issuing new requests, so a request already in flight can finish somewhat after it. Lower it in CI, where a blocking `close()` costs runner time.
</ParamField>

Returns an instance of `Evaluator`.

<CodeGroup>
  ```python simplest theme={null}
  etor = client.create_evaluator()
  ```

  ```python name and description theme={null}
  etor = client.create_evaluator(
      name="sim_iter100",
      desc="new_model_vs_sota",
      type="SMOS",
  )
  ```

  ```python full theme={null}
  etor = client.create_evaluator(
      name="sim_iter100",
      desc="new_model_vs_sota",
      type="SMOS",
      lan="en-us",
      granularity=1.0,
      num_eval=100,
      due_hours=24,
      use_loudness_normalization=True,
      auto_start=True,
      verify_batch_size=100,
      resume_upload=True,
      upload_state_path="./podonos-upload-ledger.sqlite3",
      start_timeout=1800,
  )
  ```
</CodeGroup>

### resume\_evaluator()

Resume uploads for an existing evaluation using an SDK upload ledger. Add files again in the same order as the interrupted run so the ledger can recover each file's remote object identity.

<Note>
  Resume is safety-checked against the local ledger. The ledger must contain the original evaluation contract and session configuration for `evaluation_id`; otherwise the SDK refuses to resume. The SDK restores the original session details from the ledger where available, so do not use `resume_evaluator()` to change the evaluation type, language, template, batch size, or file identities.

  During resume, completed ledger rows are reused only when the current local file still matches the recorded path, content hash, size, and upload manifest. If a completed metadata or verification row no longer matches the local file, start a fresh evaluation or remove the stale ledger row before resuming.
</Note>

<ParamField query="evaluation_id" type="string" required>
  Evaluation ID to resume.
</ParamField>

<ParamField query="upload_state_path" type="string" required>
  Path to the SDK upload ledger created by a previous upload run. It must contain the original evaluation contract for `evaluation_id`.
</ParamField>

<ParamField query="name" type="string">
  Optional session name.
</ParamField>

<ParamField query="desc" type="string">
  Optional session description.
</ParamField>

<ParamField query="type" default="NMOS" type="string">
  Evaluation type. Uses the same supported values as `create_evaluator()`.
</ParamField>

<ParamField query="lan" default="en-us" type="string">
  Language code. Uses the same supported values as `create_evaluator()`.
</ParamField>

<ParamField query="granularity" default="1.0" type="float">
  Granularity of evaluation scales.
</ParamField>

<ParamField query="num_eval" default="10" type="int">
  Number of evaluations per sample.
</ParamField>

<ParamField query="due_hours" default="12" type="int">
  Expected due time of the final report in hours.
</ParamField>

<ParamField query="use_annotation" default="False" type="bool">
  Enable annotation to collect free-form text feedback from evaluators.
</ParamField>

<ParamField query="use_loudness_normalization" default="True" type="bool">
  Enable loudness normalization for evaluation.
</ParamField>

<ParamField query="auto_start" default="False" type="bool">
  Restored from the resumed session's ledger, so the value you pass here is ignored. If the original run set it, `close()` blocks and starts the evaluation — charging your workspace balance — even when you pass `False`. The SDK logs a warning whenever the restored value disagrees with the one you passed. Requires podonos 0.46.0 or newer; earlier versions accept the parameter and silently ignore it.
</ParamField>

<ParamField query="max_upload_workers" default="20" type="int">
  Maximum number of upload worker threads.
</ParamField>

<ParamField query="verify_batch_size" default="100" type="int">
  Batch size for file verification API calls.
</ParamField>

<ParamField query="api_timeout" default="(5, 30)" type="tuple[float, float]">
  Timeout tuple for API requests, in seconds.
</ParamField>

<ParamField query="verify_timeout" default="(5, 120)" type="tuple[float, float]">
  Timeout tuple for file verification requests, in seconds.
</ParamField>

<ParamField query="upload_timeout" default="(10, 300)" type="tuple[float, float]">
  Timeout tuple for direct file upload requests, in seconds.
</ParamField>

<ParamField query="start_timeout" default="1800" type="float">
  Seconds `close()` waits when the restored `auto_start` is `True`. Unlike `auto_start` this is **not** restored from the ledger, so the value you pass here wins.
</ParamField>

```python python theme={null}
etor = client.resume_evaluator(
    evaluation_id="<EVALUATION_ID>",
    upload_state_path="./podonos-upload-ledger.sqlite3",
)
```

### create\_evaluator\_from\_template()

When you create an evaluation using a template, all the questions and options defined in the template are automatically assigned to the new evaluation. This ensures consistency and saves time by reusing pre-defined content.

<ParamField query="name" type="string" required>
  Name of this evaluation session. Required and must be non-empty.
</ParamField>

<ParamField query="template_id" type="string" required>
  The unique identifier of the template to base the new evaluation on.
</ParamField>

<ParamField query="desc" type="string">
  Description of this evaluation session. This field is for your records.
</ParamField>

<ParamField query="num_eval" default="10" type="int">
  Number of evaluations per sample.
</ParamField>

<ParamField query="use_annotation" default="False" type="bool">
  Enable annotation to collect free-form text feedback from evaluators. Cannot be used together with an `annotations` array in template JSON.
</ParamField>

<ParamField query="use_loudness_normalization" default="True" type="bool">
  Enable loudness normalization to ensure consistent audio volume levels during evaluation.
</ParamField>

<ParamField query="auto_start" default="False" type="bool">
  If `True`, `close()` waits for the uploaded files to finish processing, then starts the evaluation and **charges your workspace balance**. It raises if it cannot start. If `False`, `close()` returns as soon as the uploads are done and nothing is charged until you start the evaluation yourself in [Workspace](https://workspace.podonos.com). Requires podonos 0.46.0 or newer; earlier versions accept the parameter and silently ignore it.

  Pair it with `resume_upload=True` and an `upload_state_path`. If the start fails or times out, `resume_evaluator()` is the only way back to that evaluation; without a ledger you have to start it from Workspace instead.
</ParamField>

<ParamField query="max_upload_workers" default="20" type="int">
  Maximum number of upload worker threads.
</ParamField>

<ParamField query="verify_batch_size" default="100" type="int">
  Batch size for file verification API calls.
</ParamField>

<ParamField query="api_timeout" default="(5, 30)" type="tuple[float, float]">
  Timeout tuple for API requests, in seconds.
</ParamField>

<ParamField query="verify_timeout" default="(5, 120)" type="tuple[float, float]">
  Timeout tuple for file verification requests, in seconds.
</ParamField>

<ParamField query="upload_timeout" default="(10, 300)" type="tuple[float, float]">
  Timeout tuple for direct file upload requests, in seconds.
</ParamField>

<ParamField query="resume_upload" default="False" type="bool">
  Enable SDK-local upload ledger/resume support. The ledger records upload progress and the original evaluation contract so interrupted uploads can be resumed safely.
</ParamField>

<ParamField query="upload_state_path" default="None" type="string">
  Optional SQLite upload ledger path when `resume_upload=True`. Use the same path with `resume_evaluator()` if the upload is interrupted.
</ParamField>

<ParamField query="start_timeout" default="1800" type="float">
  Seconds `close()` waits when `auto_start=True`. Must be positive. It limits when the SDK stops issuing new requests, so a request already in flight can finish somewhat after it. Lower it in CI, where a blocking `close()` costs runner time.
</ParamField>

<CodeGroup>
  ```python python theme={null}
  etor = client.create_evaluator_from_template(
      name="Voice naturalness evaluation",
      desc="new_model_vs_competitor_model",
      num_eval=10,
      template_id="abcdef",
      use_loudness_normalization=True,
  )
  ```
</CodeGroup>

### create\_evaluator\_from\_template\_json()

Create a new evaluation using a JSON template. This allows you to define custom evaluation structures programmatically.

<ParamField query="json" type="Dict">
  Template JSON as a dictionary. Optional if `json_file` is provided.
</ParamField>

<ParamField query="json_file" type="string">
  Path to the JSON template file. Optional if `json` is provided.
</ParamField>

<ParamField query="name" type="string">
  Name of this evaluation session. Optional; if omitted, the SDK generates a name.
</ParamField>

<ParamField query="custom_type" type="string | CustomType" default="SINGLE">
  Type of evaluation. Accepts either a string or `CustomType` enum value.

  | Value         | Description                             | batch\_size | File Configuration       |
  | ------------- | --------------------------------------- | ----------- | ------------------------ |
  | `SINGLE`      | Single stimulus evaluation              | 1           | 1 stimulus               |
  | `DOUBLE`      | Double stimulus evaluation              | 2           | 2 stimuli (no reference) |
  | `SINGLE_REF`  | Reference-based comparison (CMOS style) | 2           | 1 reference + 1 stimulus |
  | `RANKING`     | Ranking evaluation                      | 2+          | 2+ stimuli               |
  | `RANKING_REF` | Ranking against a reference             | 3+          | 2+ stimuli + 1 reference |

  You can use string values (`"SINGLE"`, `"DOUBLE"`, `"SINGLE_REF"`, `"RANKING"`, `"RANKING_REF"`) or the `CustomType` enum from `podonos.common.enum`:

  ```python theme={null}
  from podonos.common.enum import CustomType

  custom_type=CustomType.SINGLE_REF
  custom_type="SINGLE_REF"
  ```
</ParamField>

<ParamField query="desc" type="string">
  Description of this evaluation session. Optional.
</ParamField>

<ParamField query="lan" default="en-us" type="string">
  Language for evaluation. See supported languages in `create_evaluator()`.
</ParamField>

<ParamField query="num_eval" default="10" type="int">
  Number of evaluations per sample.
</ParamField>

<ParamField query="use_annotation" default="False" type="bool">
  Enable annotation to collect free-form text feedback from evaluators. Cannot be used together with an `annotations` array in template JSON. When using custom annotation questions, define them in the `annotations` array instead.
</ParamField>

<ParamField query="use_loudness_normalization" default="True" type="bool">
  Enable loudness normalization to ensure consistent audio volume levels during evaluation.
</ParamField>

<ParamField query="auto_start" default="False" type="bool">
  If `True`, `close()` waits for the uploaded files to finish processing, then starts the evaluation and **charges your workspace balance**. It raises if it cannot start. If `False`, `close()` returns as soon as the uploads are done and nothing is charged until you start the evaluation yourself in [Workspace](https://workspace.podonos.com). Requires podonos 0.46.0 or newer; earlier versions accept the parameter and silently ignore it.

  Pair it with `resume_upload=True` and an `upload_state_path`. If the start fails or times out, `resume_evaluator()` is the only way back to that evaluation; without a ledger you have to start it from Workspace instead.
</ParamField>

<ParamField query="max_upload_workers" default="20" type="int">
  Maximum number of upload workers. Must be a positive integer.
</ParamField>

<ParamField query="verify_batch_size" default="100" type="int">
  Batch size for file verification API calls.
</ParamField>

<ParamField query="api_timeout" default="(5, 30)" type="tuple[float, float]">
  Timeout tuple for API requests, in seconds.
</ParamField>

<ParamField query="verify_timeout" default="(5, 120)" type="tuple[float, float]">
  Timeout tuple for file verification requests, in seconds.
</ParamField>

<ParamField query="upload_timeout" default="(10, 300)" type="tuple[float, float]">
  Timeout tuple for direct file upload requests, in seconds.
</ParamField>

<ParamField query="resume_upload" default="False" type="bool">
  Enable SDK-local upload ledger/resume support. The ledger records upload progress and the original evaluation contract so interrupted uploads can be resumed safely.
</ParamField>

<ParamField query="upload_state_path" default="None" type="string">
  Optional SQLite upload ledger path when `resume_upload=True`. Use the same path with `resume_evaluator()` if the upload is interrupted.
</ParamField>

<ParamField query="start_timeout" default="1800" type="float">
  Seconds `close()` waits when `auto_start=True`. Must be positive. It limits when the SDK stops issuing new requests, so a request already in flight can finish somewhat after it. Lower it in CI, where a blocking `close()` costs runner time.
</ParamField>

<CodeGroup>
  ```python Dictionary theme={null}
  # Using JSON dictionary
  template = {
      "questions": [
          {
              "type": "SCORED",
              "question": "How natural is the voice?",
              "description": "Rate the quality of the voice",
              "options": [
                  {"label_text": "Excellent"},
                  {"label_text": "Good"},
                  {"label_text": "Fair"},
                  {"label_text": "Poor"},
                  {"label_text": "Bad"},
              ],
          }
      ]
  }

  evaluator = client.create_evaluator_from_template_json(
      json=template,
      name="Quality Test",
      custom_type="SINGLE",
  )
  ```

  ```python JSON File theme={null}
  # Using JSON file
  evaluator = client.create_evaluator_from_template_json(
      json_file="./template.json",
      name="Quality Test",
      custom_type="DOUBLE",
  )
  ```

  ```python SINGLE_REF (CMOS style) theme={null}
  from podonos import File
  from podonos.common.enum import CustomType

  # CMOS-style reference comparison
  template = {
      "questions": [
          {
              "type": "SCORED",
              "question": "How similar is the synthesized audio to the reference?",
              "options": [
                  {"label_text": "Identical"},
                  {"label_text": "Very similar"},
                  {"label_text": "Somewhat similar"},
                  {"label_text": "Different"},
                  {"label_text": "Completely different"},
              ],
          }
      ]
  }

  evaluator = client.create_evaluator_from_template_json(
      json=template,
      name="CMOS Style Evaluation",
      custom_type=CustomType.SINGLE_REF,  # or "SINGLE_REF"
  )

  # Add files: 1 stimulus + 1 reference
  evaluator.add_files(
      file0=File(path="synthesized.wav", model_tag="tts"),
      file1=File(path="reference.wav", model_tag="human", is_ref=True),
  )
  ```

  ```python RANKING_REF theme={null}
  from podonos import File

  # Ranking templates take COMPARISON questions only.
  template = {
      "questions": [
          {
              "type": "COMPARISON",
              "question": "Which sample is closer to the reference?",
              "anchor_label": {"label_text": {"left": "is closer", "right": "is closer"}},
          }
      ]
  }

  evaluator = client.create_evaluator_from_template_json(
      json=template,
      name="Ranking Against a Reference",
      custom_type="RANKING_REF",
  )

  # One set per script: 2+ stimuli plus exactly one reference.
  # The reference keeps the same model_tag in every set.
  for i in range(3):
      evaluator.add_ranking_set([
          File(path=f"model_a_{i}.wav", model_tag="model_a"),
          File(path=f"model_b_{i}.wav", model_tag="model_b"),
          File(path=f"model_c_{i}.wav", model_tag="model_c"),
          File(path=f"reference_{i}.wav", model_tag="reference", is_ref=True),
      ])
  ```
</CodeGroup>

Returns an instance of `Evaluator`.

Here's the JSON template for reference:

<Tabs>
  <Tab title="Question">
    `Question`: Represents the main question posed to evaluators about the audio being assessed. It guides evaluators on the specific aspect of the audio they should focus on during the evaluation.

    | Parameter        | Description                                                              | Required             | Notes                                                                 |
    | ---------------- | ------------------------------------------------------------------------ | -------------------- | --------------------------------------------------------------------- |
    | `type`           | Type of question. Options: `SCORED`, `NON_SCORED`, `COMPARISON`          | Yes                  | Determines the structure and requirements of the question             |
    | `question`       | The main question text                                                   | Yes                  | Must be provided for all question types                               |
    | `description`    | Additional details or context for the question                           | No                   | Optional for all question types                                       |
    | `options`        | List of possible options. Only for `SCORED` and `NON_SCORED` types       | Conditional          | Must have between 1 and 9 options for `SCORED` and `NON_SCORED` types |
    | `scale`          | Scale for comparison. Only for `COMPARISON` type                         | Conditional          | Must be an integer between 2 and 9 for `COMPARISON` type              |
    | `allow_multiple` | Allows multiple selections. Only for `NON_SCORED` type                   | Yes for `NON_SCORED` | Enables multiple choice selection                                     |
    | `has_other`      | Includes an "Other" option. Only for `NON_SCORED` type                   | No                   | Adds an option for evaluators to specify an unlisted choice           |
    | `has_none`       | Includes a "None" option. Only for `NON_SCORED` type                     | No                   | Adds an option for evaluators to select none of the listed choices    |
    | `related_model`  | Related model for the question. Only for `Double` Evaluation type.       | Conditional          | Select which model the question is related to.                        |
    | `anchor_label`   | Labels for the ends of the comparison scale. Only for `COMPARISON` type. | Conditional          | Provides context for what each end of the scale represents.           |

    <Note>
      Important Notes:

      * `SCORED` and `NON_SCORED` questions can have a maximum of 9 options.
      * `NON_SCORED` questions must specify `allow_multiple`.
      * `COMPARISON` type questions must have a scale between 2 and 9.
      * `related_model` consists of `ALL`, `MODEL_A` and `MODEL_B`. Default is `ALL`. The `related_model` is only used for the question (not for instructions).
    </Note>
  </Tab>

  <Tab title="Option">
    `Option`: Represents a possible answer or choice in a question. It provides evaluators with a range of options to choose from.

    | Parameter        | Description                                                                                       | Required | Notes                                                                  |
    | ---------------- | ------------------------------------------------------------------------------------------------- | -------- | ---------------------------------------------------------------------- |
    | `label_text`     | The text displayed for the option                                                                 | Yes      | This is the text shown to the evaluator                                |
    | `reference_file` | A reference audio file for the option. It helps the evaluator understand the quality of the audio | No       | This file serves as a benchmark and is saved in the evaluation results |

    <Note>
      For `options` in `SCORED` and `NON_SCORED` questions:

      * `score` is automatically generated only for `SCORED` questions. If there are 5 options, the first option in the list receives a score of 5, the second option receives a score of 4, and so on, down to a score of 1 for the last option.
      * `order` is the index of the option in the list, starting from 0.
    </Note>
  </Tab>

  <Tab title="Anchor Label">
    `Anchor Label`: Only for `COMPARISON` type. Provides context for what each end of the scale represents, enhancing evaluator understanding.

    | Parameter    | Description                                                                                           | Required | Notes                                                   |
    | ------------ | ----------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------- |
    | `title`      | The title of the anchor label                                                                         | No       | Provides a clear and concise title for the anchor label |
    | `label_text` | The text displayed for the anchor label. `left` and `right` should be provided for `COMPARISON` type. | Yes      | This is the text shown to the evaluator                 |

    <Note>
      For `COMPARISON` type:

      * `left` and `right` should be provided for `label_text`.
      * `title` is optional.

      ```json theme={null}
      {
        "anchor_label": {
          "title": "Clarity",
          "label_text": {
            "left": "Much worse",
            "right": "Much better"
          }
        }
      }
      ```
    </Note>
  </Tab>

  <Tab title="Instruction">
    `Instruction`: Provides key guidance to evaluators on how to approach the evaluation.

    | Parameter         | Description                                                      | Required | Notes                                                                                                                           |
    | ----------------- | ---------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------- |
    | `type`            | Type of instruction. Options: `DO`, `WARNING`, `DONT`, `EXAMPLE` | Yes      | Guides evaluators on how to approach the evaluation                                                                             |
    | `instruction`     | The main instruction text                                        | Yes      | Provides key guidance to evaluators                                                                                             |
    | `description`     | Additional details or context for the instruction                | No       | Optional for all instruction types                                                                                              |
    | `reference_files` | A list of reference files to set a benchmark for evaluators      | No       | Up to 3 items. Each item must be a dictionary with `path` and `type`, where `type` is one of `audio`, `reference`, or `target`. |

    <Note>
      Additional Notes:

      * `Instruction` is optional.
      * `description` and `reference_files` are optional but can provide valuable context.
      * `reference_files` uses this shape: `[{"path": "./sample.wav", "type": "audio"}]`.
    </Note>
  </Tab>

  <Tab title="Annotation">
    `Annotation`: Collects free-form text feedback from evaluators. Defined in the `annotations` array.

    | Parameter       | Description                           | Required    | Notes                                                                 |
    | --------------- | ------------------------------------- | ----------- | --------------------------------------------------------------------- |
    | `type`          | Must be `ANNOTATION`                  | Yes         | Identifies this as an annotation question                             |
    | `question`      | The question text shown to evaluators | Yes         | Cannot be empty                                                       |
    | `related_model` | Which audio the annotation applies to | Conditional | `ALL` for single stimulus, `MODEL_A` or `MODEL_B` for double stimulus |
    | `description`   | Additional guidance for evaluators    | No          | Optional hint for what feedback to provide                            |

    <Note>
      Important Notes:

      * The `annotations` array is separate from the `questions` array.
      * For single stimulus evaluations, use `related_model: "ALL"` or omit it.
      * For double stimulus evaluations, `related_model` must be `"MODEL_A"` or `"MODEL_B"`.
      * Cannot use `annotations` array together with `use_annotation=True` parameter.
    </Note>

    **Example:**

    ```json theme={null}
    {
      "questions": [...],
      "annotations": [
        {
          "type": "ANNOTATION",
          "question": "Describe any issues you noticed",
          "related_model": "ALL",
          "description": "e.g., noise, pronunciation errors"
        }
      ]
    }
    ```
  </Tab>
</Tabs>

### flash\_eval()

Run automatic evaluation on an audio file and return a `FlashEvalResult`.

<ParamField query="file_path" type="string" required>
  Path to the audio file to evaluate.
</ParamField>

<ParamField query="language" type="string">
  Language code for model routing. Currently available: `en-us`, `es-es`. When omitted, the default `en-us` model is used. Ignored when `category="noise_quality"`.
</ParamField>

<ParamField query="category" type="string">
  Evaluation category. Defaults to `naturalness` when omitted. Currently available: `naturalness`, `noise_quality`.
</ParamField>

Returns a `FlashEvalResult` with these fields:

| Field           | Description                                         |
| --------------- | --------------------------------------------------- |
| `naturalness`   | Naturalness score when `category="naturalness"`     |
| `noise_quality` | Noise quality score when `category="noise_quality"` |
| `file`          | The evaluated `File` object                         |
| `id`            | Optional evaluation ID                              |
| `message`       | Optional service message                            |

```python python theme={null}
result = client.flash_eval(file_path="path/to/audio.wav")
print(result.naturalness)

spanish_result = client.flash_eval(file_path="path/to/audio_es.wav", language="es-es")
noise_result = client.flash_eval(file_path="path/to/audio.wav", category="noise_quality")
print(noise_result.noise_quality)
```

### get\_evaluation\_list()

Returns a JSON containing all your evaluations, each with its current status and progress.

<CodeGroup>
  ```python python theme={null}
  evaluations = client.get_evaluation_list()
  print(evaluations)
  ```
</CodeGroup>

The output JSON looks like:

```json theme={null}
[
  {
    "id": "<UUID>",
    "title": "How natural my synthetic voices are",
    "internal_name": null,
    "description": "Used latest internal model. Epoch 10, alpha 0.1",
    "batch_size": 1,
    "status": "ACTIVE",
    "progress": 52.5,
    "started_time": "2024-06-25T02:03:11.005Z",
    "ended_time": null,
    "created_time": "2024-06-25T01:40:43.429Z",
    "updated_time": "2024-06-26T13:21:34.801Z"
  }
]
```

`progress`, `started_time` and `ended_time` are reported by podonos 0.47.0 or newer. `started_time` and `ended_time` are `null` until the evaluation starts and ends, and all three are `null` when the service does not report them, so check for `null` before comparing.

<Note>
  An evaluation is finished when `status` is `"COMPLETED"`, never when `progress` reaches 100. `progress` is capped at 90 while the evaluation is running and stays at 90 during report review, so it cannot tell "almost done" from "done". Use `status` to decide, and `progress` for display only.

  To check a single evaluation instead of listing the whole workspace, see [get\_evaluation\_progress()](#get-evaluation-progress).
</Note>

### get\_evaluation\_progress()

Returns the current status and progress of one evaluation, without listing the rest of the workspace. Requires podonos 0.47.0 or newer.

<ParamField query="evaluation_id" type="string" required>
  Evaluation id. See `get_evaluation_list()`, or `Evaluator.get_evaluation_id()` on an evaluator you created.
</ParamField>

Returns an `EvaluationProgress` with these fields. Call `.to_dict()` for the same values as a JSON-style dictionary.

| Field          | Type                 | Description                                                                            |
| -------------- | -------------------- | -------------------------------------------------------------------------------------- |
| `id`           | `str`                | The evaluation id                                                                      |
| `status`       | `str`                | One of `DRAFT`, `SUBMITTED`, `SCHEDULED`, `ACTIVE`, `COMPLETED`, `CANCELED`, `DELETED` |
| `progress`     | `float`              | 0 to 100. Capped at 90 while running; 100 only once `status` is `COMPLETED`            |
| `started_time` | `datetime` or `None` | UTC. `None` until the evaluation starts                                                |
| `ended_time`   | `datetime` or `None` | UTC. `None` until the evaluation ends                                                  |

<CodeGroup>
  ```python poll until complete theme={null}
  import time
  from podonos.errors import EvaluationNotFoundError

  pending = {etor.get_evaluation_id() for etor in my_evaluators}
  results = {}

  while pending:
      for evaluation_id in list(pending):
          try:
              progress = client.get_evaluation_progress(evaluation_id)
          except EvaluationNotFoundError:
              pending.discard(evaluation_id)   # not in this workspace any more
              continue
          if progress.status == "COMPLETED":
              results[evaluation_id] = client.get_stats_json_by_id(evaluation_id)
              pending.discard(evaluation_id)
          elif progress.status in {"CANCELED", "DELETED"}:
              pending.discard(evaluation_id)
      time.sleep(max(len(pending), 10))
  ```

  ```python single evaluation theme={null}
  progress = client.get_evaluation_progress("<EVALUATION_ID>")
  print(progress.status, progress.progress)   # e.g. ACTIVE 52.5
  print(progress.to_dict())
  ```
</CodeGroup>

<Note>
  Stop polling on `COMPLETED`, `CANCELED` or `DELETED`. A deleted or hidden evaluation still reports its real status here, even though `get_evaluation_list()` no longer lists it.

  Decide on `status`, never on `progress >= 100`, for the reason given under `get_evaluation_list()`.
</Note>

<Warning>
  The service allows **60 requests per minute per API key**, counted across all evaluation ids rather than per evaluation. Polling `N` evaluations therefore needs an interval of at least `N` seconds; the example above sleeps `max(N, 10)`. The SDK retries a rate-limited request with backoff, and those retries spend the same budget, so a sustained overrun surfaces as a `podonos.common.exception.HTTPError` with status 429.

  Two other exceptions are possible:

  * `podonos.errors.EvaluationNotFoundError` when the id is not in your API key's workspace. The service does not distinguish an id that does not exist from one that belongs to another workspace.
  * `podonos.common.exception.HTTPError` for any other failure, including a key that is missing or revoked, and a service that does not serve evaluation progress. In the latter case, `get_evaluation_list()` still reports progress for every evaluation in the workspace.
</Warning>

### get\_eval\_template\_info()

Gets detailed information about an evaluation template by its ID.

<ParamField query="template_id" type="string">
  The unique identifier of the evaluation template to retrieve information for.
</ParamField>

Returns a Python dictionary containing detailed template information.

<CodeGroup>
  ```python python theme={null}
  template_info = client.get_eval_template_info("abcdef")
  print(template_info)
  ```
</CodeGroup>

The returned dictionary has this shape:

```python theme={null}
{
  "id": "<UUID>",
  "code": "abcdef",
  "title": "Voice Quality Assessment",
  "description": "Template for evaluating voice naturalness and quality",
  "language": Language.ENGLISH_AMERICAN,
  "eval_type": "Single",
  "created_time": datetime.datetime(2024, 6, 25, 1, 40, 43, 429000, tzinfo=datetime.timezone.utc),
  "updated_time": datetime.datetime(2024, 6, 26, 13, 21, 34, 801000, tzinfo=datetime.timezone.utc)
}
```

| Field          | Description                                                                              |
| -------------- | ---------------------------------------------------------------------------------------- |
| `id`           | Unique identifier (UUID) of the template                                                 |
| `code`         | Template code used for identification                                                    |
| `title`        | Display name of the template                                                             |
| `description`  | Detailed description of the template's purpose                                           |
| `language`     | `Language` enum value from the template, such as `Language.ENGLISH_AMERICAN` for `en-us` |
| `eval_type`    | Type of evaluation: `Single`, `Double`, or `Triple`                                      |
| `created_time` | `datetime.datetime` value for when the template was created                              |
| `updated_time` | `datetime.datetime` value for when the template was last modified                        |

### get\_stats\_json\_by\_id()

Returns a list of JSONs containing the statistics of each stimulus for the evaluation referenced by the `id`.

<ParamField query="evaluation_id" type="string">
  Evaluation id. See `get_evaluation_list()`.
</ParamField>

<ParamField query="group_by" default="question" type="string">
  Group by criteria. Options are "question", "script", or "model". Default is "question". Note that "script" and "model" are only available for single-question evaluations.
</ParamField>

<CodeGroup>
  ```python python theme={null}
  evaluations = client.get_evaluation_list()
  for eval in evaluations:
      stats = client.get_stats_json_by_id(eval['id'], group_by='question')
      print(stats)
  ```
</CodeGroup>

| Field       | Description                                                  | SCORED | NON\_SCORED |
| ----------- | ------------------------------------------------------------ | ------ | ----------- |
| `frequency` | List of score counts: `{"score": number, "count": number}[]` | ✓      | -           |
| `mean`      | Average score                                                | ✓      | -           |
| `median`    | Median score                                                 | ✓      | -           |
| `std`       | Standard deviation                                           | ✓      | -           |
| `sem`       | Standard error of the mean                                   | ✓      | -           |
| `ci_95`     | 95% confidence interval                                      | ✓      | -           |
| `options`   | Each option name as key with integer value                   | ✓      | ✓           |
| `OTHER`     | The number of evaluators who selected "Other"                | ✓      | ✓           |

<Note>
  For NON\_SCORED questions:

  * The integer value is the number of evaluators who selected the option.
  * All options are included in the response regardless of their value
</Note>

<Tabs>
  <Tab title="Single">
    You can get the statistics of each question by calling `get_stats_json_by_id()` with `group_by` set to `question`, `script`, or `model`.

    <CodeGroup>
      ```json question theme={null}
      {
        "question": string,
        "description": string,
        "order": int,
        "responses": [
          {
            "name": string,
            "model_tag": string,
            "tags": string[],
            "type": "A" | "B" | "REF",
            "script": string | null,
            "frequency": [
              {
                "score": number,
                "count": number
              }
            ],
            "mean": float | null, // null if the question is not SCORED
            "median": float | null, // null if the question is not SCORED
            "std": float | null, // null if the question is not SCORED
            "sem": float | null, // null if the question is not SCORED
            "ci_95": float | null, // null if the question is not SCORED
          }
        ]
      }
      ```

      ```json script theme={null}
      {
        "script": string,
        "responses": [
          {
            "question": string,
            "description": string,
            "order": int,
            "responses": [
              {
                "name": string,
                "model_tag": string,
                "tags": string[],
                "type": "A" | "B" | "REF",
                "script": string | null,
                "frequency": [
                  {
                    "score": number,
                    "count": number
                  }
                ],
                "mean": float | null, // null if the question is not SCORED
                "median": float | null, // null if the question is not SCORED
                "std": float | null, // null if the question is not SCORED
                "sem": float | null, // null if the question is not SCORED
                "ci_95": float | null, // null if the question is not SCORED
              },
              ...
            ]
          },
          ...
        ]
      }
      ```

      ```json model theme={null}
      {
        "model": string,
        "responses": [
          {
            "question": string,
            "description": string,
            "order": int,
            "responses": [
              {
                "name": string,
                "model_tag": string,
                "tags": string[],
                "type": "A" | "B" | "REF",
                "script": string | null,
                "frequency": [
                  {
                    "score": number,
                    "count": number
                  }
                ],
                "mean": float | null, // null if the question is not SCORED
                "median": float | null, // null if the question is not SCORED
                "std": float | null, // null if the question is not SCORED
                "sem": float | null, // null if the question is not SCORED
                "ci_95": float | null, // null if the question is not SCORED
              },
              ...
            ]
          },
          ...
        ]
      }
      ```
    </CodeGroup>
  </Tab>

  <Tab title="Double (including CSMOS)">
    The `group_by` is set to `question` by default. Other options (`script`, `model`) cannot be used.

    <CodeGroup>
      ```json question theme={null}
      {
        "question": string,
        "description": string,
        "order": int,
        "responses": [
          {
            "targets": [
              {
                "name": string,
                "model_tag": string,
                "tags": string[],
                "type": "A" | "B" | "REF",
                "script": string | null,
              }
            ],
            "frequency": [
              {
                "score": number,
                "count": number
              }
            ],
            "mean": float | null, // null if the question is not SCORED
            "median": float | null, // null if the question is not SCORED
            "std": float | null, // null if the question is not SCORED
            "sem": float | null, // null if the question is not SCORED
            "ci_95": float | null, // null if the question is not SCORED
          }
        ]
      }
      ```
    </CodeGroup>
  </Tab>
</Tabs>

### download\_evaluation\_files\_by\_evaluation\_id()

Download all files associated with a specific evaluation, identified by its `evaluation_id`, from the Podonos evaluation service. It saves these files to a specified directory on the local file system and generates a metadata file describing the downloaded files.
Return a string indicating the status of the download operation. This could be a success message or an error message if the download fails.

<ParamField query="evaluation_id" type="string">
  Evaluation id. See `get_evaluation_list()`.
</ParamField>

<ParamField query="output_dir" type="string">
  The directory path where the downloaded files will be saved. This should be a valid path on the local file system where the user has write permissions.
</ParamField>

<CodeGroup>
  ```python example theme={null}
  client.download_evaluation_files_by_evaluation_id(
    evaluation_id="12345",
    output_dir="./output",
  )
  ```

  ```json metadata.json theme={null}
  {
    "files": [
      {
        "file_path": "./output/model1/4c318c0fd5a492d3e20cb8142ac5344b.wav",
        "original_name": "file1.wav",
        "model_tag": "model1",
        "tags": ["tag1", "tag2"]
      }
    ]
  }
  ```
</CodeGroup>

| Field           | Description                                                                                                                                             |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `file_path`     | The path produced by joining `output_dir` with `{model_tag}/{hashed_file_name}`. It can be relative or absolute depending on the `output_dir` you pass. |
| `original_name` | The original name of the file before downloading.                                                                                                       |
| `model_tag`     | The model tag associated with the file, used for categorization.                                                                                        |
| `tags`          | A list of tags associated with the file, providing additional context or categorization.                                                                |

<Note>
  File Naming Convention:

  Each downloaded file is saved in the format `{output_dir}/{model_tag}/{file_name}`. This means that files are organized into subdirectories named after their `model_tag`, and the original file name is hashed formatted.
</Note>

## File

A class representing one file, used for adding files in `Evaluator`.

<ParamField query="path" type="string" required>
  Path to the file to evaluate. For audio files, we support `wav`, `mp3`, and `flac` formats.
</ParamField>

<ParamField query="model_tag" type="string" required>
  Name of your model (e.g., `WhisperTTS`) or any unique name (e.g., `human`).
</ParamField>

<ParamField query="tags" type="list[string]">
  A list of string tags for the file designated by `path`. You can use this field as properties of the file such as `original`, `synthesized`, `tom`, `maria`, and so on. Later you can look up or group files with particular tags in the output report.
</ParamField>

<ParamField query="script" type="string">
  Text script of the input audio file.
</ParamField>

<ParamField query="is_ref" default="False" type="bool">
  True if this file works as a reference in a comparative evaluation.
</ParamField>

<ParamField query="meta_data" type="Dict[str, str | int | float | bool | None]">
  Optional metadata dictionary for the file. Keys must be strings, and values must be JSON-primitive values.
</ParamField>

<ParamField query="script_tags" type="list[string]">
  Optional tags for the script. These are useful for ranking and script-level grouping.
</ParamField>

```python python theme={null}
file = File(
    path="/path/to/speech.wav",
    model_tag="my_model",
    tags=["synthesized", "male"],
    script="hello there",
    meta_data={"speaker_id": "spk-001"},
    script_tags=["greeting"],
)
```

## Evaluator

`Evaluator` manages a single type of evaluation.

### add\_file()

Add one file to evaluate in a single evaluation question. For a single file evaluation like `NMOS`, one file to evaluate is added.

<ParamField query="file" type="File" required>
  Input `File`. This field is required if `type` is `NMOS`, `QMOS`, `P808`, or `CUSTOM_SINGLE`.
</ParamField>

<CodeGroup>
  ```python NMOS theme={null}
  etor.add_file(
      file=File(
          path="/path/to/speech_0_0.wav",
          model_tag="my_model",
          tags=["synthesized", "male", "ver1234"],
      )
  )
  ```

  ```python P808 theme={null}
  etor.add_file(
      file=File(
          path="/path/to/speech_0_0.wav",
          model_tag="my_model",
          tags=["synthesized", "female", "ver1234"],
      )
  )
  ```
</CodeGroup>

### add\_files()

Add multiple files for evaluations that require comparison.

<ParamField query="file0" type="File" required>
  First input file.
</ParamField>

<ParamField query="file1" type="File" required>
  Second input file.
</ParamField>

<ParamField query="file2" type="File">
  Optional third input file. Used for `CSMOS` evaluations with one reference and two stimuli.
</ParamField>

The file requirements depend on evaluation type:

| Type                    | File requirements                                                         |
| ----------------------- | ------------------------------------------------------------------------- |
| `PREF`, `CUSTOM_DOUBLE` | Two stimulus files, in any order                                          |
| `SMOS`                  | Two stimulus files, in any order                                          |
| `CMOS`                  | One stimulus file and one reference file, in any order                    |
| `CSMOS`                 | Two stimulus files in any order, followed by one reference file (`file2`) |

<Note>
  **You do not need to keep a consistent argument order, and you should not shuffle it yourself.** The SDK normalizes what it stores: stimuli are sorted by `model_tag` (numeric-aware and case-insensitive) and the reference is stored last. The same model therefore occupies the same position in every group, which is what lets the platform attribute scores back to it.

  Presentation order is randomized per evaluator regardless — see [Bias Minimization](/docs/reliability/bias-minimization).

  The one positional rule left: for `CSMOS` the reference must be passed as `file2`.
</Note>

<CodeGroup>
  ```python SMOS theme={null}
  file0 = File(path="/path/to/speech0.wav", model_tag="human", tags=["original", "male"])
  file1 = File(path="/path/to/speech1.wav", model_tag="my_model", tags=["synthesized", "male", "ver1234"])
  etor.add_files(file0=file0, file1=file1)
  ```

  ```python CSMOS theme={null}
  reference = File(path="/path/to/reference.wav", model_tag="human", is_ref=True)
  model_a = File(path="/path/to/model_a.wav", model_tag="model_a")
  model_b = File(path="/path/to/model_b.wav", model_tag="model_b")
  etor.add_files(file0=model_a, file1=model_b, file2=reference)
  ```
</CodeGroup>

### add\_ranking\_set()

Add one ranking set for a `RANKING` or `RANKING_REF` evaluation.

<ParamField query="files" type="list[File]" required>
  Candidate files for one ranking group. For `RANKING_REF`, include exactly one file with `is_ref=True`; its position in the list does not matter.
</ParamField>

Constraints enforced across calls:

* All groups must have the same number of files.
* The order of stimulus `model_tag` must be identical across groups.
* Within a group, every `model_tag` must be unique, case-insensitively. The reference is included in this check, so it cannot reuse a candidate's tag.
* `RANKING`: every file must be a stimulus. Setting `is_ref=True` raises.
* `RANKING_REF`: exactly one file with `is_ref=True` and at least two stimuli. The reference's `model_tag` must be the same in every group.

<CodeGroup>
  ```python RANKING theme={null}
  etor = client.create_evaluator(type="RANKING", name="Ranking test")

  etor.add_ranking_set([
      File(path="/path/to/model_a_001.wav", model_tag="model_a", script_tags=["script_001"]),
      File(path="/path/to/model_b_001.wav", model_tag="model_b", script_tags=["script_001"]),
      File(path="/path/to/model_c_001.wav", model_tag="model_c", script_tags=["script_001"]),
  ])
  ```

  ```python RANKING_REF theme={null}
  etor = client.create_evaluator(type="RANKING_REF", name="Ranking against a reference")

  etor.add_ranking_set([
      File(path="/path/to/model_a_001.wav", model_tag="model_a", script_tags=["script_001"]),
      File(path="/path/to/model_b_001.wav", model_tag="model_b", script_tags=["script_001"]),
      File(path="/path/to/model_c_001.wav", model_tag="model_c", script_tags=["script_001"]),
      File(path="/path/to/reference_001.wav", model_tag="reference", script_tags=["script_001"], is_ref=True),
  ])
  ```
</CodeGroup>

<Note>
  `model_tag` accepts letters, digits, hyphens and underscores only. `"Provider A"` raises; use `"Provider_A"`.
</Note>

### close()

Close the evaluation session. Once this function is called, all the evaluation files will be sent to the Podonos evaluation service, the files will go through a series of processing, and delivered to evaluators.

Returns a JSON object containing the uploading status.

```python python theme={null}
status = etor.close()
```

With `auto_start=True` on podonos 0.46.0 or newer, `close()` does not return once the uploads finish. It waits for the files to become processable, starts the evaluation — **charging your workspace balance** — and returns only after the evaluation is running.

<Warning>
  With `auto_start=True`, `close()` can block for up to `start_timeout` (30 minutes by default) and can raise where it previously always returned. Two different exception types are possible, and they do not share a base class:

  * `podonos.common.exception.HTTPError` when the service refuses the start, for example when the workspace balance is insufficient. The message carries the server's error code.
  * the builtin `TimeoutError` when `start_timeout` elapses first. `except HTTPError` will **not** catch this one.

  A timeout does not mean the evaluation did not start. The SDK reports it as unconfirmed because a start whose response was lost may well have succeeded and been charged. Check the evaluation in [Workspace](https://workspace.podonos.com) before running it again, rather than paying twice.
</Warning>

```python auto_start theme={null}
etor = client.create_evaluator(
    name="nightly_regression",
    type="NMOS",
    auto_start=True,
    start_timeout=600,          # 10 minutes is usually plenty in CI
    resume_upload=True,
    upload_state_path="./podonos-upload-ledger.sqlite3",
)
etor.add_file(File(path="audio1.wav", model_tag="model_a"))
etor.add_file(File(path="audio2.wav", model_tag="model_b"))

etor.close()   # blocks through processing, then starts the evaluation
```

Calling `close()` with no files added raises a `ValueError`.