espnet2.train.iterable_dataset.IterableESPnetDataset

source

class espnet2.train.iterable_dataset.IterableESPnetDataset(path_name_type_list: Collection[Tuple[str, str, str]], preprocess: Callable[[str, Dict[str, ndarray]], Dict[str, ndarray]] | None = None, float_dtype: str = 'float32', int_dtype: str = 'long', key_file: str | List | None = None, preprocess_prefix: str | None = None)

Bases: IterableDataset

Pytorch Dataset class for ESPNet.

This class represents an iterable dataset for handling various data types used in ESPNet training. It supports loading and preprocessing data from specified file types and yields data in a structured format.

path_name_type_list

A collection of tuples containing paths, names, and types of the data files.

• Type: Collection[Tuple[str, str, str]]

preprocess (Optional[Callable[[str, Dict[str, np.ndarray]],

Dict[str, np.ndarray]]]): An optional preprocessing function to apply to the data.

float_dtype

The desired float data type (default: “float32”).

• Type: str

int_dtype

The desired integer data type (default: “long”).

• Type: str

key_file

An optional file or list containing keys for data access.

• Type: Optional[Union[str, List]]

preprocess_prefix

An optional prefix for the preprocessing function.

• Type: Optional[str]

debug_info

A dictionary for storing debug information about the dataset.

• Type: Dict[str, Tuple[str, str]]

non_iterable_dataset

An instance of ESPnetDataset for handling non-iterable data types.

• Type: Optional[ESPnetDataset]

apply_utt2category

A flag indicating whether to apply utterance-to-category mapping.

• Type: bool

• Parameters:

  • path_name_type_list (Collection *[*Tuple *[*str , str , str ] ]) – A list of tuples where each tuple contains (path, name, type).
  • **(Optional[Callable[****[**str (preprocess) – Dict[str, np.ndarray]]]): A function for preprocessing data.
  • Dict**[**str – Dict[str, np.ndarray]]]): A function for preprocessing data.
  • np.ndarray**]****]** – Dict[str, np.ndarray]]]): A function for preprocessing data.

:param : Dict[str, np.ndarray]]]): A function for preprocessing data. :param float_dtype: Desired data type for floating-point values. :type float_dtype: str :param int_dtype: Desired data type for integer values. :type int_dtype: str :param key_file: File or list of keys for data

retrieval.

• Parameters:preprocess_prefix (Optional *[*str ]) – Prefix to prepend to keys for preprocessing.
• Raises:
  • ValueError – If path_name_type_list is empty.
  • RuntimeError – If duplicate names are found or if keys are mismatched.

####### Examples

>>> dataset = IterableESPnetDataset([('wav.scp', 'input', 'sound'),
...                                  ('token_int', 'output', 'text_int')],
...                                )
>>> for uid, data in dataset:
...     print(data)
{'input': per_utt_array, 'output': per_utt_array}

has_name(name) → bool

Checks if the given name exists in the dataset.

This method verifies if a specific name is present in the dataset’s internal debug information, which tracks the paths and types of the data entries.

• Parameters:name (str) – The name to check for existence in the dataset.
• Returns: True if the name exists in the dataset, False otherwise.
• Return type: bool

####### Examples

>>> dataset = IterableESPnetDataset([('wav.scp', 'input', 'sound')])
>>> dataset.has_name('input')
True
>>> dataset.has_name('output')
False

names() → Tuple[str, ...]

Iterable dataset module.

This module contains classes and functions for creating and managing iterable datasets for ESPNet.

Functions: : load_kaldi(input): Load Kaldi format data.

Constants: : DATA_TYPES (dict): A dictionary mapping data types to loading functions.

Classes: : IterableESPnetDataset: A Pytorch Dataset class for ESPNet. SplicedIterableESPnetDataset: A data iterator that is spliced from multiple IterableESPnetDataset.