espnet2.asr.encoder.beats_encoder.gelu

source

espnet2.asr.encoder.beats_encoder.gelu(x: Tensor) → Tensor

Applies the Gaussian Error Linear Unit (GELU) activation function.

The GELU activation function is defined mathematically as: GELU(x) = x * P(X <= x), where P is the cumulative distribution function of a standard normal distribution. It has been found to perform better than traditional activation functions such as ReLU and tanh in certain neural network architectures.

This implementation casts the input tensor to float32 before applying the GELU function to ensure numerical stability, then casts it back to the original data type.

• Parameters:x (torch.Tensor) – Input tensor of any shape. The tensor will be cast to float32 for computation.
• Returns: Output tensor after applying the GELU activation function, : with the same shape and type as the input tensor.
• Return type: torch.Tensor

Examples

>>> import torch
>>> input_tensor = torch.tensor([-1.0, 0.0, 1.0, 2.0])
>>> output_tensor = gelu(input_tensor)
>>> print(output_tensor)
tensor([-0.1587,  0.0000,  0.8413,  1.7615])

NOTE

This function uses PyTorch’s built-in GELU function for performance and accuracy.