Skip to main content

espnet2.schedulers.warmup_reducelronplateau.WarmupReduceLROnPlateau

About 6 min

espnet2.schedulers.warmup_reducelronplateau.WarmupReduceLROnPlateau

source

class espnet2.schedulers.warmup_reducelronplateau.WarmupReduceLROnPlateau(optimizer: Optimizer, warmup_steps: int | float = 25000, mode='min', factor=0.1, patience=10, threshold=0.0001, threshold_mode='rel', cooldown=0, min_lr=0, eps=1e-08, verbose=False)

Bases: AbsBatchStepScheduler, AbsValEpochStepScheduler

The WarmupReduceLROnPlateau scheduler.

This scheduler combines the functionality of WarmupLR and ReduceLROnPlateau:

WarmupLR: : lr = optimizer.lr * warmup_step ** 0.5 : * min(step ** -0.5, step * warmup_step ** -1.5)

WarmupReduceLROnPlateau: : if step <= warmup_step: : lr = optimizer.lr * warmup_step ** 0.5 : * min(step ** -0.5, step * warmup_step ** -1.5)
else: : lr = ( : optimizer.lr * factor if no improvement for a ‘patience’ number of epochs else optimizer.lr <br/> )

Note that the maximum lr equals to optimizer.lr in this scheduler.

warmup_steps

Number of steps for the warmup phase.

  • Type: Union[int, float]

step_num

Current step number.

  • Type: int

lr_scale

Scaling factor for learning rate during warmup.

  • Type: float

base_lrs

Initial learning rates for each parameter group.

  • Type: list

factor

Factor by which the learning rate will be reduced.

  • Type: float

optimizer

The optimizer to be used.

  • Type: torch.optim.Optimizer

min_lrs

Minimum learning rates for each parameter group.

  • Type: list

patience

Number of epochs with no improvement after which learning rate will be reduced.

  • Type: int

verbose

If True, prints a message to stdout for each learning rate reduction.

  • Type: bool

cooldown

Number of epochs to wait before resuming normal operation after learning rate has been reduced.

  • Type: int

mode

One of {‘min’, ‘max’}. In ‘min’ mode, lr is reduced when the quantity monitored has stopped decreasing; in ‘max’ mode it is reduced when the quantity monitored has stopped increasing.

  • Type: str

threshold

Threshold for measuring the new optimum, to only focus on significant changes.

  • Type: float

threshold

_mode

One of {‘rel’, ‘abs’}. In ‘rel’ mode, dynamic threshold is a relative change, in ‘abs’ mode it is an absolute change.

  • Type: str

eps

Minimal decay applied to learning rate.

  • Type: float

last_epoch

The index of the last epoch.

  • Type: int

best

The best value seen so far.

  • Type: float

num_bad_epochs

Number of bad epochs.

  • Type: int

mode

_worse

The worse value for the chosen mode.

  • Type: float

cooldown

_counter

Counter for cooldown period.

  • Type: int

_last_lr

Last learning rates.

  • Type: list

  • Parameters:

    • optimizer (torch.optim.Optimizer) – The optimizer for which to schedule the learning rate.
    • warmup_steps (Union *[*int , float ] , optional) – Number of steps for warmup. Defaults to 25000.
    • mode (str , optional) – One of {‘min’, ‘max’}. Defaults to ‘min’.
    • factor (float , optional) – Factor by which the learning rate will be reduced. Defaults to 0.1.
    • patience (int , optional) – Number of epochs with no improvement after which learning rate will be reduced. Defaults to 10.
    • threshold (float , optional) – Threshold for measuring new optimum. Defaults to 1e-4.
    • threshold_mode (str , optional) – One of {‘rel’, ‘abs’}. Defaults to ‘rel’.
    • cooldown (int , optional) – Number of epochs to wait before resuming normal operation after learning rate has been reduced. Defaults to 0.
    • min_lr (Union *[*int , float , list ] , optional) – Minimum learning rate for each parameter group. Defaults to 0.
    • eps (float , optional) – Minimal decay applied to learning rate. Defaults to 1e-8.
    • verbose (bool , optional) – If True, prints a message to stdout for each learning rate reduction. Defaults to False.

############# Examples

Create an optimizer

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

Initialize the scheduler

scheduler = WarmupReduceLROnPlateau(optimizer, warmup_steps=1000)

Step the scheduler

for epoch in range(num_epochs):

… training code …

scheduler.step(metrics=validation_metric, epoch=epoch)

  • Raises:
    • ValueError – If factor is >= 1.0 or if the number of min_lrs does not
    • match the number of parameter groups in the optimizer. –

property in_cooldown

is_better(a, best)

Determines if the current metric is better than the best recorded.

The comparison is based on the mode (‘min’ or ‘max’) and the threshold mode (‘rel’ or ‘abs’) set during the initialization of the scheduler.

  • Parameters:
    • a (float) – The current metric to compare.
    • best (float) – The best recorded metric.
  • Returns: True if the current metric is better than the best recorded metric, False otherwise.
  • Return type: bool

############# Examples

>>> scheduler = WarmupReduceLROnPlateau(optimizer, mode='min',
...                                       threshold_mode='rel',
...                                       threshold=0.1)
>>> scheduler.is_better(0.8, 0.9)
False
>>> scheduler.is_better(0.7, 0.9)
True

load_state_dict(state_dict)

Load the state dictionary for the scheduler.

This method updates the internal state of the scheduler using the provided state dictionary. It also reinitializes the internal parameters related to the comparison mode and threshold settings.

  • Parameters:state_dict (dict) – A dictionary containing the state of the scheduler. This should include all necessary attributes that were saved previously using state_dict().

############# Examples

>>> scheduler = WarmupReduceLROnPlateau(optimizer)
>>> state = scheduler.state_dict()
>>> scheduler.load_state_dict(state)

####### NOTE It is essential that the state_dict provided has been created using the same version of the WarmupReduceLROnPlateau class to ensure compatibility.

state_dict()

Returns the state dictionary of the scheduler.

This method returns a dictionary containing the internal state of the WarmupReduceLROnPlateau scheduler, which can be used for saving and loading the state of the scheduler.

The state dictionary includes all attributes of the class except for the optimizer, which is not serialized.

  • Returns: A dictionary containing the state of the scheduler.
  • Return type: dict

############# Examples

>>> scheduler = WarmupReduceLROnPlateau(optimizer, warmup_steps=5000)
>>> state = scheduler.state_dict()
>>> print(state)
{'warmup_steps': 5000, 'step_num': 0, 'lr_scale': 0.0004472135954999579,
 'base_lrs': [...], 'factor': 0.1, 'min_lrs': [...], ...}

####### NOTE The returned state_dict can be used with the load_state_dict method to restore the scheduler state.

step(metrics=None, epoch=None)

The WarmupReduceLROnPlateau scheduler.

This scheduler combines the functionality of WarmupLR and ReduceLROnPlateau:

WarmupLR: : lr = optimizer.lr * warmup_step ** 0.5 : * min(step ** -0.5, step * warmup_step ** -1.5)

WarmupReduceLROnPlateau: : if step <= warmup_step: : lr = optimizer.lr * warmup_step ** 0.5 : * min(step ** -0.5, step * warmup_step ** -1.5)
else: : lr = ( : optimizer.lr * factor if no improvement for a ‘patience’ number of epochs else optimizer.lr <br/> )

Note that the maximum lr equals to optimizer.lr in this scheduler.

warmup_steps

Number of steps for warmup phase.

  • Type: Union[int, float]

step_num

Current step number.

  • Type: int

lr_scale

Scale factor for learning rate during warmup.

  • Type: float

base_lrs

Initial learning rates for each parameter group.

  • Type: list

factor

Factor by which the learning rate will be reduced.

  • Type: float

optimizer

The optimizer to adjust the learning rate for.

  • Type: torch.optim.Optimizer

min_lrs

Minimum learning rates for each parameter group.

  • Type: list

patience

Number of epochs with no improvement after which learning rate will be reduced.

  • Type: int

verbose

If True, prints a message to stdout for each learning rate adjustment.

  • Type: bool

cooldown

Number of epochs to wait before resuming normal operation after lr has been reduced.

  • Type: int

threshold

Threshold for measuring the new optimum.

  • Type: float

threshold

_mode

One of “rel” or “abs”, to specify whether threshold is a relative or absolute change.

  • Type: str

eps

Minimal change in learning rate to be considered significant.

  • Type: float

last_epoch

Last epoch number.

  • Type: int

best

Best metric observed during training.

  • Type: float

num_bad_epochs

Counter for the number of bad epochs.

  • Type: int

mode

_worse

The worse value for the chosen mode.

  • Type: float

cooldown

_counter

Counter for cooldown periods.

  • Type: int

_last_lr

Last learning rates for each parameter group.

  • Type: list

  • Parameters:

    • optimizer (torch.optim.Optimizer) – The optimizer to adjust the learning rate for.
    • warmup_steps (Union *[*int , float ] , optional) – Number of steps for warmup phase. Default is 25000.
    • mode (str , optional) – One of {‘min’, ‘max’}. Default is ‘min’.
    • factor (float , optional) – Factor by which the learning rate will be reduced. Default is 0.1.
    • patience (int , optional) – Number of epochs with no improvement after which learning rate will be reduced. Default is 10.
    • threshold (float , optional) – Threshold for measuring the new optimum. Default is 1e-4.
    • threshold_mode (str , optional) – One of {‘rel’, ‘abs’}. Default is ‘rel’.
    • cooldown (int , optional) – Number of epochs to wait before resuming normal operation after lr has been reduced. Default is 0.
    • min_lr (Union *[*int , float , list ] , optional) – Minimum learning rate. Default is 0.
    • eps (float , optional) – Minimal change in learning rate to be considered significant. Default is 1e-8.
    • verbose (bool , optional) – If True, prints a message to stdout for each learning rate adjustment. Default is False.

  • Returns: None

############# Examples

>>> optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
>>> scheduler = WarmupReduceLROnPlateau(optimizer, warmup_steps=5000)
>>> for epoch in range(100):
>>>     train(...)
>>>     val_metrics = validate(...)
>>>     scheduler.step(val_metrics, epoch)

####### NOTE The metrics parameter in the step method can be a float value representing the validation metric. If no metrics are provided, it will perform warmup adjustments based on the step count.

Copyright © 2024 ESPnet Community. All rights reserved.