espnet3.parallel.base_runner.concatenate_shard_files

source

espnet3.parallel.base_runner.concatenate_shard_files(shard_dirs: Sequence[Path], relative_name: str, out_path: Path) → bool

Concatenate shard-local text files into a single output file.

Looks for relative_name under each directory in shard_dirs and appends the contents of every existing fragment to out_path in the given order. If no fragment exists, this function removes out_path when present and returns False.

• Parameters:
  • shard_dirs – Ordered shard directories to scan for file fragments.
  • relative_name – Relative path of the fragment file inside each shard directory.
  • out_path – Destination path for the concatenated file.
• Returns:True if at least one shard fragment was found and written, False otherwise.
• Return type: bool
• Raises:OSError – If the output parent directory cannot be created.

Example

>>> from pathlib import Path
>>> shard_dirs = [Path("output/split.0"), Path("output/split.1")]
>>> found = concatenate_shard_files(
...     shard_dirs, "hyp.scp", Path("output/hyp.scp")
... )
>>> print(found)  # True if at least one shard contained hyp.scp
True