espnet2.optimizers.optim_groups.add_optimizer_hooks

source

espnet2.optimizers.optim_groups.add_optimizer_hooks(model, bias_weight_decay=False, normalization_weight_decay=False)

Set zero weight decay for certain model parameters.

This function configures the weight decay for parameters in the given model. It sets weight_decay=0.0 for parameters that meet any of the following criteria:

• Parameters in model.no_weight_decay.
• Parameters with the attribute _no_weight_decay==True.
• Bias parameters if bias_weight_decay is False.
• Normalization parameters if normalization_weight_decay is False.

For more information on weight decay behavior, refer to the following discussion: https://discuss.pytorch.org/t/weight-decay-only-for-weights-of-nn-linear-and-nn-conv/114348

• Parameters:
  • model (nn.Module) – The model whose parameters are to be configured.
  • bias_weight_decay (bool , optional) – If False, bias parameters will have zero weight decay. Defaults to False.
  • normalization_weight_decay (bool , optional) – If False, normalization parameters will have zero weight decay. Defaults to False.

Examples

>>> import torch.nn as nn
>>> model = nn.Sequential(nn.Linear(10, 5), nn.BatchNorm1d(5))
>>> add_optimizer_hooks(model, bias_weight_decay=True, normalization_weight_decay=False)

NOTE

This function modifies the parameters of the model in place.