espnet2.enh.layers.tcn.GlobalLayerNorm

source

class espnet2.enh.layers.tcn.GlobalLayerNorm(channel_size, shape='BDT')

Bases: Module

Global Layer Normalization (gLN).

This class implements global layer normalization, which normalizes the input across all dimensions, helping to stabilize the learning process and improve convergence. It scales and shifts the normalized output using learnable parameters gamma and beta.

gamma

Learnable parameter for scaling the normalized output.

beta

Learnable parameter for shifting the normalized output.

shape

Defines the shape of the input tensor. Options are “BDT” (Batch, Depth, Time) or “BTD” (Batch, Time, Depth).

• Parameters:
  • channel_size – Number of channels to normalize.
  • shape – Shape of the input tensor, either “BDT” or “BTD”.
• Raises:AssertionError – If the shape is not “BDT” or “BTD”.

######### Examples

>>> gLN = GlobalLayerNorm(channel_size=64, shape="BDT")
>>> input_tensor = torch.randn(32, 64, 100)  # [M, N, K]
>>> output_tensor = gLN(input_tensor)
>>> output_tensor.shape
torch.Size([32, 64, 100])

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(y)

Global Layer Normalization (gLN).

This layer normalizes the input across all dimensions except the batch dimension, applying a learnable scale and shift to the normalized values.

gamma

Learnable scale parameter.

• Type: torch.nn.Parameter

beta

Learnable shift parameter.

• Type: torch.nn.Parameter

shape

Defines the input shape arrangement, either ‘BDT’ or ‘BTD’.

• Type: str

• Parameters:

  • channel_size (int) – The number of channels in the input.
  • shape (str) – The shape of the input, either ‘BDT’ (Batch, Channel, Time) or ‘BTD’ (Batch, Time, Channel). Defaults to ‘BDT’.

• Returns: Normalized output tensor with the same shape as the input.

• Return type: gLN_y

######### Examples

>>> layer_norm = GlobalLayerNorm(channel_size=64)
>>> input_tensor = torch.randn(32, 64, 100)  # [M, N, K]
>>> output_tensor = layer_norm(input_tensor)
>>> print(output_tensor.shape)
torch.Size([32, 64, 100])  # Output shape remains the same

NOTE

The layer applies the normalization based on the mean and variance calculated across the specified dimensions, ensuring stability and faster convergence during training.

• Raises:AssertionError – If the shape is not ‘BDT’ or ‘BTD’.

reset_parameters()

Global Layer Normalization (gLN).

This class implements global layer normalization, which normalizes the input across both spatial dimensions and batches, making it suitable for tasks where the input distribution varies across these dimensions.

gamma

Learnable scale parameter of shape (1, channel_size, 1).

beta

Learnable shift parameter of shape (1, channel_size, 1).

shape

The shape of the input tensor, either “BDT” or “BTD”.

• Parameters:
  • channel_size – The number of channels in the input tensor.
  • shape – The shape of the input tensor, either “BDT” or “BTD”.
• Raises:AssertionError – If the shape is not one of “BDT” or “BTD”.

######### Examples

>>> import torch
>>> gLN = GlobalLayerNorm(channel_size=10)
>>> input_tensor = torch.randn(32, 10, 100)  # [M, N, K]
>>> output_tensor = gLN(input_tensor)
>>> output_tensor.shape
torch.Size([32, 10, 100])

NOTE

The forward pass computes the mean and variance of the input tensor and normalizes it using the learnable parameters gamma and beta.