​
Intro

The Podonos SDK has three major components: Client, Evaluator, and File.

​
Three Key Components

​
Client

Client manages multiple evaluations and can create multiple Evaluators. Each evaluator corresponds to one types of evaluation (e.g., naturalness, preferences). So if you want to evaluate your inputs along multiple dimensions, you will create multiple evaluators from one client like: 
etor_natural = client.create_evaluator(
    name='My first naturalness evaluation',
    desc='My new speech generation model',
    type='NMOS', lan='en-us', num_eval=7
)
There are key parameters:
  • name: Name of the evaluation.
  • desc: Detailed description of the evaluation.
  • type: One of the evaluation types. For details, see type in create_evaluator().
  • lan: Language to be evaluated. Real human who fluently speaks the language and lives in the country will evaluate.
  • num_eval: The number of people who evaluates one file. The higher this number is, the more reliable the evaluation will be.
  • use_annotation: This is for uncovering the evaluation details.
  • use_loudness_normalization: When set to true, audio files will be volume-normalized before evaluation.
  • max_upload_workers: The maximum number of upload workers.
You can also get the whole list of evaluations you created so far by calling get_evaluation_list().
Python
eval_list = client.get_evaluation_list()
print(eval_list)
In addition, you can get the detailed statistics of one evaluation:
Python
eval_detail_list = client.get_stats_json_by_id(evaluation_id='<EVALUATION_ID>')
print(eval_detail_list)
For more details, see get_stats_json_by_id().

​
Evaluator

Evaluator object manages the details of one evaluation type. With this object, you can add files to evaluate. 
etor.add_file(file=myfile))

​
File

File represents one file and its details. 
File(path='/path/to/1.mp3', model_tag='my_new_model',
     tags=['male', 'american', 'narrative'])
Here are the key parameters:
  • path: Path to the file to evaluate.
  • model_tag: Name of your model (or any unique name). In the evluation report, you will see the statistics grouped by this model name.
  • tags: List of multiple tags strings. In the evaluation report, you can see the statistics grouped by one or more tags.
For more details, see File.

​
Tags

In the File above, you see the two parameters: model_tag and tags. The model_tag may be the name of your AI model our any other unique name. tags are any unique strings that describes the attributes of the file. They may include training parameters, genders, voice types, ages, regions, or any.