espnet2.asr_transducer.normalization.BasicNorm

source

class espnet2.asr_transducer.normalization.BasicNorm(normalized_shape: int, eps: float = 0.25)

Bases: Module

BasicNorm module for performing basic normalization.

This module applies basic normalization to input tensors, which involves scaling the input based on the mean of the squares of its elements for each feature. It is useful for stabilizing training in neural networks.

Reference: : https://github.com/k2-fsa/icefall/pull/288

• Parameters:
  • normalized_shape (int) – Expected size of the input features.
  • eps (float , optional) – Value added to the denominator for numerical stability. Default is 0.25.

####### Examples

>>> import torch
>>> basic_norm = BasicNorm(normalized_shape=10, eps=0.1)
>>> input_tensor = torch.randn(2, 5, 10)  # (B, T, D_hidden)
>>> output_tensor = basic_norm(input_tensor)
>>> output_tensor.shape
torch.Size([2, 5, 10])

Construct a BasicNorm object.

forward(x: Tensor) → Tensor

Compute basic normalization.

This method applies basic normalization to the input tensor x by computing the scaling factor based on the mean of the squares of the input elements and applying it to the input tensor.

• Parameters:
  • x – Input sequences. Shape (B, T, D_hidden), where:
  • size ( - B is the batch)

:param : :param - T is the sequence length: :param : :param - D_hidden is the dimensionality of the hidden states.:

• Returns: Output sequences after applying basic normalization. Shape (B, T, D_hidden).
• Return type: Tensor

####### Examples

>>> norm = BasicNorm(normalized_shape=128)
>>> input_tensor = torch.randn(32, 10, 128)  # Batch of 32, seq length 10
>>> output_tensor = norm(input_tensor)
>>> output_tensor.shape
torch.Size([32, 10, 128])

NOTE

The normalization is performed by scaling the input tensor based on the computed scales which are derived from the mean of the squares of the input tensor.