espnet2.hubert.espnet_model.TorchAudioHubertPretrainModel

source

class espnet2.hubert.espnet_model.TorchAudioHubertPretrainModel(vocab_size: int, token_list: Tuple[str, ...] | List[str], frontend: AbsFrontend | None, specaug: AbsSpecAug | None, normalize: AbsNormalize | None, preencoder: AbsPreEncoder | None, encoder: AbsEncoder, ignore_id: int = -1, **kwargs)

Bases: AbsESPnetModel

TorchAudio Hubert Pretrain model

Initialize internal Module state, shared by both nn.Module and ScriptModule.

collect_feats(speech: Tensor, speech_lengths: Tensor, text: Tensor, text_lengths: Tensor, **kwargs) → Dict[str, Tensor]

encode(speech: Tensor, speech_lengths: Tensor, y_pad: Tensor, y_pad_length: Tensor) → Tuple[Tensor, Tensor]

Frontend + Encoder. Note that this method is used by asr_inference.py

• Parameters:
  • speech – (Batch, Length, …)
  • speech_lengths – (Batch, )
  • y_pad – (Batch, Length, …)
  • y_pad_length – (Batch, )

forward(speech: Tensor, speech_lengths: Tensor, text: Tensor, text_lengths: Tensor, **kwargs) → Tuple[Tensor, Dict[str, Tensor], Tensor]

Frontend + Encoder + Calc loss

• Parameters:
  • speech – (Batch, Length, …)
  • speech_lengths – (Batch, )
  • text – (Batch, Length)
  • text_lengths – (Batch,)
  • kwargs – “utt_id” is among the input.