ESPnet3 Dataset Configuration

This page explains the dataset section used across training, inference, and metrics configs.

Important

ESPnet3 dataset config is centered on DataOrganizer plus dataset reference entries. Do not assume each split directly instantiates a _target_ dataset class.

Basic shape

Most recipes use DataOrganizer and define train, valid, and test entries under dataset.

dataset:
  _target_: espnet3.components.data.data_organizer.DataOrganizer
  recipe_dir: ${recipe_dir}
  train:
    - data_src_args:
        split: train
  valid:
    - data_src_args:
        split: valid
  test:
    - name: test
      data_src_args:
        split: test

What each dataset entry contains

Each dataset entry may contain:

• name
• data_src
• data_src_args
• transform

The main rule is:

• data_src_args goes to Dataset(...)
• name and transform stay in organizer space

data_src

data_src tells ESPnet3 where the dataset module comes from.

Supported forms:

Recipe tag

data_src: mini_an4/asr

This resolves to:

egs3.mini_an4.asr.dataset

Explicit module path

data_src: egs3.mini_an4.asr.dataset

Omit data_src

If data_src is omitted, ESPnet3 loads:

${recipe_dir}/dataset/__init__.py

This is the normal pattern for recipe-local dataset code.

data_src_args

data_src_args is forwarded directly to the exported Dataset class.

data_src_args:
  split: test-clean
  recipe_dir: ${recipe_dir}
  source_dir: ${dataset_dir}

This becomes:

Dataset(split="test-clean", recipe_dir=recipe_dir, source_dir=dataset_dir)

Split meanings

• train is used by collect_stats and train
• valid is used by collect_stats and validation during training
• test is used by infer and may also help measure

For inference, each dataset.test[].name becomes the output subdirectory under ${inference_dir}.

preprocessor

dataset.preprocessor is optional. When set, it is instantiated by DataOrganizer and applied after dataset loading.

Relationship to create_dataset

create_dataset is separate from dataset.

• create_dataset prepares manifests or raw assets
• dataset describes how later stages load them

If you want the full dataset module contract, see Dataset references and builders.

Related pages

Training Config

See how dataset config fits into `training.yaml`.

Inference Config

See how `dataset.test` is used by inference.

Dataset References

Read the full `DataOrganizer`, `data_src`, and builder contract.