espnet2.asr_transducer.normalization.ScaleNorm

source

class espnet2.asr_transducer.normalization.ScaleNorm(normalized_shape: int, eps: float = 1e-05)

Bases: Module

ScaleNorm module definition.

This module performs scale normalization on the input tensor. The scaling is based on the L2 norm of the input, ensuring that the output has a consistent scale. This can be useful in various neural network architectures where normalization of activations is required.

Reference: : https://arxiv.org/pdf/1910.05895.pdf

• Parameters:
  • normalized_shape – Expected size of the input tensor for normalization.
  • eps – A small value added to the denominator for numerical stability (default: 1e-5).

####### Examples

>>> scale_norm = ScaleNorm(normalized_shape=128)
>>> input_tensor = torch.randn(32, 10, 128)  # (B, T, D_hidden)
>>> output_tensor = scale_norm(input_tensor)
>>> print(output_tensor.shape)
torch.Size([32, 10, 128])

• Returns: Output sequences with the same shape as the input, scaled based on the computed norm.

Construct a ScaleNorm object.

#

forward(x

: Tensor) → Tensor

ScaleNorm module definition.

Reference: https://arxiv.org/pdf/1910.05895.pdf

• Parameters:
  • normalized_shape – Expected size of the input tensor.
  • eps – Value added to the denominator for numerical stability (default: 1e-5).

scale

A learnable parameter representing the scaling factor.

forward(x

torch.Tensor) -> torch.Tensor: Computes the scale normalization for the input tensor.

####### Examples

>>> import torch
>>> scale_norm = ScaleNorm(normalized_shape=10)
>>> input_tensor = torch.randn(5, 2, 10)  # Batch size of 5, seq length of 2
>>> output_tensor = scale_norm(input_tensor)
>>> print(output_tensor.shape)
torch.Size([5, 2, 10])