> ## Documentation Index
> Fetch the complete documentation index at: https://podonos.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# SDK Template Usage
> You can create evaluations quickly by using templates.
## Create Evaluations from Templates
For developers, the SDK provides two powerful ways to create evaluations programmatically using templates:
### Option 1: Using Template ID
Navigate to the "Templates" Tab on the **Workspace**, select the template you
want to use, and copy the **Template ID**.
Initialize with your API key. For details, refer to [Get API Key](docs/apikey.mdx)
```python theme={null}
client = podonos.init("")
```
Use the Template ID to create a new evaluation.
```python theme={null}
evaluator = client.create_evaluator_from_template(
name="Evaluate the english voice",
desc="model_1_and_model_2",
num_eval=10,
template_id="HTGRqU" # Evaluation template ID from the Workspace
)
```
Add files with relevant metadata to the evaluator.
```python single theme={null}
for i in range(60):
file = File(path=f"path/speech_{i}.mp3", tags=["man", "noisy"],
model_tag="Model 1")
evaluator.add_file(file)
```
```python double theme={null}
for i in range(60):
file0 = File(path=f"path2/speech_{i}.mp3", tags=["man", "gentle"],
model_tag="Model 1")
file1 = File(path=f"path2/speech_{i}.mp3", tags=["woman", "mild"],
model_tag="Model 2")
evaluator.add_files(file0, file1)
```
Close the evaluator to finalize the setup.
```python theme={null}
evaluator.close()
```
### Option 2: Using Template JSON
You can create evaluations by defining the template structure either directly in your code using a JSON dictionary or by loading it from a JSON file.
#### Intro
A template consists of three main sections:
1. **Questions Section**: Contains the core evaluation questions
* `SCORED`: Rating scale questions (e.g., 1-5 Likert scale)
* `NON_SCORED`: Multiple/Single choice questions
* `COMPARISON`: Comparison questions (for double evaluations)
2. **Instructions Section** (Optional): Contains guidance messages to the evaluators.
* These messages help ensure that evaluators understand how to conduct the evaluation properly. Providing clear and appropriate guidance can significantly enhance the quality of the evaluation results.
* Use one of the following types: `DO`, `WARNING`, `DONT`, `EXAMPLE`
3. **Annotations Section** (Optional): Collects free-form text feedback from evaluators.
* Allows evaluators to provide detailed written feedback beyond standard ratings.
* Use `related_model` to specify which audio the annotation applies to.
#### Types
```json theme={null}
{
"type": "SCORED",
"question": "Please evaluate the overall quality of the audio",
"description": "You can ignore the wrong pronunciation of the audio",
"options": [
{"label_text": "Excellent"},
{"label_text": "Good"},
{
"label_text": "Fair",
"reference_file": "path/to/reference/file" # Helps the evaluator to select the quality of the audio
},
{"label_text": "Poor"},
{"label_text": "Bad"},
]
}
```
For `options` in `SCORED` questions: - `score` is automatically generated only
for `SCORED` questions. If there are 5 options, the first option in the list
receives a score of 5, the second option receives a score of 4, and so on,
down to a score of 1 for the last option. - `order` is the index of the option
in the list, starting from 0. - For more detailed explanations, please refer
to the [reference
documentation](reference#create-evaluator-from-template-json).
```json theme={null}
{
"type": "NON_SCORED",
"question": "Please select all the bad audio characteristics",
"description": "Select only the audio characteristics that are distracting or undesirable.",
"options": [
{"label_text": "Background Noise"},
{
"label_text": "Echo",
"reference_file": "path/to/reference/file" # Can tell why this audio has echo
},
{"label_text": "Distortion"}
],
"allow_multiple": true, # If the evaluator can select multiple options
"has_other": true # If the evaluator can select other options
}
```
For `options` in `NON_SCORED` questions:
* `allow_multiple` is a boolean value that determines if the evaluator can select multiple options.
* `has_other` is a boolean value that determines if the evaluator can select other options.
* For more detailed explanations, please refer to the [reference documentation](reference#create-evaluator-from-template-json).
```json theme={null}
{
"type": "COMPARISON",
"question": "Listen to the speech/audio samples and rate which one you prefer.",
"description": "Ignore the background noise and audio quality",
"scale": 5 # Required. 5-point comparison scale
"anchor_label": {
"title": "Preferences", # Optional
"label_text": {
"left": "is better", # The prefix (A) is automatically added to the left label
"right": "is better" # The prefix (B) is automatically added to the right label
}
}
}
```
For `scale` in `COMPARISON` questions:
* `scale` is an required integer value between 2 and 9.
* `anchor_label` should be provided for giving context to the evaluator.
* For more detailed explanations, please refer to the [reference documentation](reference#create-evaluator-from-template-json).
```json theme={null}
{
"type": "EXAMPLE",
"instruction": "Use the provided reference file to assess the quality of the audio.",
"description": "Audio quality is not considered for the evaluation",
"reference_files": [
{
"path": "./aws/speech_1_Joanna.mp3",
"type": "audio"
},
{
"path": "./aws/reference_1.mp3",
"type": "reference"
},
{
"path": "./aws/target_1.mp3",
"type": "target"
}
]
}
```
For `reference_files` in `Instruction`:
* `reference_files` is an optional array that helps the evaluator to select the quality of the audio.
* Each file must have a `path` and a `type`. The `type` must be one of: `audio`, `reference`, or `target`.
* Use the `EXAMPLE` type when you want to provide users with more examples to better understand the evaluation context.
* For more detailed explanations, please refer to the [reference documentation](reference#create-evaluator-from-template-json).
Annotation questions collect free-form text feedback from evaluators beyond standard rating scales.
```json theme={null}
{
"type": "ANNOTATION",
"question": "Describe any issues you noticed in the audio",
"description": "e.g., noise, pronunciation errors, unnatural intonation",
"related_model": "ALL"
}
```
For double stimulus evaluations, specify which audio the annotation applies to:
```json theme={null}
{
"annotations": [
{
"type": "ANNOTATION",
"question": "Describe issues in Audio A",
"related_model": "MODEL_A"
},
{
"type": "ANNOTATION",
"question": "Describe issues in Audio B",
"related_model": "MODEL_B"
}
]
}
```
For `annotations`:
* Annotations are defined in a separate `annotations` array, not in `questions`.
* `related_model` must be `"ALL"` for single stimulus, or `"MODEL_A"`/`"MODEL_B"` for double stimulus.
* You can also enable annotations automatically with `use_annotation=True` parameter instead of defining the array.
* Cannot use both `use_annotation=True` and `annotations` array together.
* For more detailed explanations, please refer to the [reference documentation](reference#create-evaluator-from-template-json).
#### Example
Here's a step-by-step guide to create an evaluation using the template JSON:
Define your template as a Python dictionary or save it as a JSON file:
```python Single Dict theme={null}
# Single evaluation template example
TEMPLATE = {
"instructions": [
{
"type": "DO",
"instruction": "Use the provided reference file to assess the quality of the audio.",
"reference_files": [
{
"path": "./aws/speech_1_Joanna.mp3",
"type": "audio"
},
{
"path": "./aws/speech_1_Joanna.mp3",
"type": "reference"
}
]
},
{
"type": "WARNING",
"instruction": "Please evaluate only the audio's natural pronunciation",
"description": "Just focus on the audio's naturalness, the background noise or audio quality is not considered for the evaluation",
},
],
"questions": [
{
"type": "SCORED",
"question": "Please evaluate the overall quality of the audio",
"description": "You can ignore the wrong pronunciation of the audio",
"options": [
{"label_text": "Excellent"},
{"label_text": "Good"},
{"label_text": "Fair", "reference_file": "path/to/reference/file"},
{"label_text": "Poor"},
{"label_text": "Bad"},
]
},
{
"type": "NON_SCORED",
"question": "Select all the bad audio characteristics",
"description": "Select only the audio characteristics that are distracting or undesirable.",
"options": [
{"label_text": "Background Noise"},
{"label_text": "Echo", "reference_file": "path/to/reference/file"},
{"label_text": "Distortion"}
],
"allow_multiple": true,
"has_other": true
}
],
"annotations": [
{
"type": "ANNOTATION",
"question": "If you rated poorly, please describe the issues",
"description": "e.g., noise, pronunciation errors, unnatural intonation",
"related_model": "ALL"
}
]
}
```
```python Double Dict theme={null}
# Double evaluation template example
TEMPLATE = {
"instructions": [
{
"type": "EXAMPLE",
"instruction": "Evaluation Guidelines",
"description": "Important points to consider when evaluating audio",
"reference_files": [
{
"path": "./aws/speech_1_Joanna.mp3",
"type": "audio"
},
{
"path": "./aws/speech_1_reference.mp3",
"type": "reference"
},
{
"path": "./aws/speech_1_target.mp3",
"type": "target"
}
]
},
{
"type": "DONT",
"instruction": "Please do not consider background noise when evaluating the audio's clarity.",
"description": "Just focus on the audio's naturalness, the background noise or audio quality is not considered for the evaluation"
}
],
"questions": [
{
"type": "COMPARISON",
"question": "Which audio sample is clearer?",
"description": "Focus on the clarity of the speech",
"scale": 5,
"anchor_label": {
"title": "Clarity",
"label_text": {
"left": "is clearer",
"right": "is clearer"
}
}
},
{
"type": "NON_SCORED",
"question": "Why did you choose the sample?",
"options": [
{"label_text": "Good pronunciation"},
{"label_text": "Naturalness"},
{"label_text": "Emotionality"}
],
"allow_multiple": false,
"has_other": true,
"related_model": "MODEL_A"
}
],
"annotations": [
{
"type": "ANNOTATION",
"question": "Describe any issues in Audio A",
"related_model": "MODEL_A"
},
{
"type": "ANNOTATION",
"question": "Describe any issues in Audio B",
"related_model": "MODEL_B"
}
]
}
```
```json Single JSON theme={null}
{
"instructions": [
{
"type": "DO",
"instruction": "Use the provided reference file to assess the quality of the audio.",
"reference_files": [
{
"path": "./aws/speech_1_Joanna.mp3",
"type": "audio"
},
{
"path": "./aws/speech_1_Joanna.mp3",
"type": "reference"
}
]
},
{
"type": "WARNING",
"instruction": "Please evaluate only the audio's natural pronunciation",
"description": "Just focus on the audio's naturalness, the background noise or audio quality is not considered for the evaluation"
}
],
"questions": [
{
"type": "SCORED",
"question": "Please evaluate the overall quality of the audio",
"description": "You can ignore the wrong pronunciation of the audio",
"options": [
{"label_text": "Excellent"},
{"label_text": "Good"},
{"label_text": "Fair", "reference_file": "path/to/reference/file"},
{"label_text": "Poor"},
{"label_text": "Bad"}
]
},
{
"type": "NON_SCORED",
"question": "Select all the bad audio characteristics",
"description": "Select only the audio characteristics that are distracting or undesirable.",
"options": [
{"label_text": "Background Noise"},
{"label_text": "Echo", "reference_file": "path/to/reference/file"},
{"label_text": "Distortion"}
],
"allow_multiple": true,
"has_other": true
}
],
"annotations": [
{
"type": "ANNOTATION",
"question": "If you rated poorly, please describe the issues",
"description": "e.g., noise, pronunciation errors, unnatural intonation",
"related_model": "ALL"
}
]
}
```
```json Double JSON theme={null}
{
"instructions": [
{
"type": "EXAMPLE",
"instruction": "Evaluation Guidelines",
"description": "Important points to consider when evaluating audio",
"reference_files": [
{
"path": "./aws/speech_1_Joanna.mp3",
"type": "audio"
},
{
"path": "./aws/speech_1_reference.mp3",
"type": "reference"
},
{
"path": "./aws/speech_1_target.mp3",
"type": "target"
}
]
},
{
"type": "DONT",
"instruction": "Please do not consider background noise when evaluating the audio's clarity.",
"description": "Just focus on the audio's naturalness, the background noise or audio quality is not considered for the evaluation"
}
],
"questions": [
{
"type": "COMPARISON",
"question": "Which audio sample is clearer?",
"description": "Focus on the clarity of the speech",
"scale": 5,
"anchor_label": {
"title": "Clarity",
"label_text": {
"left": "is clearer",
"right": "is clearer"
}
}
},
{
"type": "NON_SCORED",
"question": "Why did you choose the sample?",
"options": [
{ "label_text": "Good pronunciation" },
{ "label_text": "Naturalness" },
{ "label_text": "Emotionality" }
],
"allow_multiple": false,
"has_other": true,
"related_model": "MODEL_A"
}
],
"annotations": [
{
"type": "ANNOTATION",
"question": "Describe any issues in Audio A",
"related_model": "MODEL_A"
},
{
"type": "ANNOTATION",
"question": "Describe any issues in Audio B",
"related_model": "MODEL_B"
}
]
}
```
When using a JSON file:
* Save the file with `.json` extension
* Ensure proper JSON formatting
* Use UTF-8 encoding
For single and double evaluations (evaluating one or two files at a time):
```python Single Dict theme={null}
# Single evaluation template example
evaluator = client.create_evaluator_from_template_json(
json=SINGLE_TEMPLATE,
name="Single Audio Quality Test",
desc="model_a_and_model_b",
num_eval=10,
custom_type="SINGLE" # Specify SINGLE type
)
# Add single files
for i in range(3):
file = File(path=f"./audio/speech\_{i}.mp3",
model_tag="Model A",
tags=["clean", "male"])
evaluator.add_file(file)
```
```python Double Dict theme={null}
# Double evaluation template example
evaluator = client.create_evaluator_from_template_json(
json=DOUBLE_TEMPLATE,
name="Double Audio Quality Test",
desc="model_a_and_model_b",
num_eval=10,
custom_type="DOUBLE" # Specify DOUBLE type
)
# Add file pairs
for i in range(3):
file_a = File(path=f"./audio/model_a_{i}.mp3",
model_tag="Model A",
tags=["enhanced"])
file_b = File(path=f"./audio/model_b_{i}.mp3",
model_tag="Model B",
tags=["baseline"])
evaluator.add_files(file_a, file_b)
```
```python Single JSON theme={null}
# Single evaluation template example
evaluator = client.create_evaluator_from_template_json(
json_file="path/to/single_evaluation.json",
name="Single Audio Quality Test",
custom_type="SINGLE" # Specify SINGLE type
)
# Add single files
for i in range(3):
file = File(path=f"./audio/speech_{i}.mp3",
model_tag="Model A",
tags=["clean", "male"])
evaluator.add_file(file)
```
```python Double JSON theme={null}
# Double evaluation template example
evaluator = client.create_evaluator_from_template_json(
json_file="path/to/double_evaluation.json",
name="Audio Comparison Test",
custom_type="DOUBLE" # Specify DOUBLE type
)
# Add file pairs
for i in range(3):
file_a = File(path=f"./audio/model_a_{i}.mp3",
model_tag="Model A",
tags=["enhanced"])
file_b = File(path=f"./audio/model_b_{i}.mp3",
model_tag="Model B",
tags=["baseline"])
evaluator.add_files(file_a, file_b)
```
For single evaluations:
* Use `add_file()` method for adding individual files
* Each file is evaluated independently
* Avoid using COMPARISON type questions
For double evaluations:
* Use `add_files()` method to add pairs of files
* Files are always evaluated in pairs
* COMPARISON type questions are supported
* Consider using clear model\_tag for distinguishing each file
Close the evaluator to finalize the setup:
```python theme={null}
evaluator.close()
```
Tip: Using the SDK is ideal for integrating evaluations into automated
workflows.