Results for "warmup"

Gradually increasing learning rate at training start to avoid divergence.

Warmup is like stretching before a workout. Just as you wouldn’t want to start running at full speed without warming up your muscles, a machine learning model benefits from gradually increasing its learning rate at the beginning of training. This helps the model avoid making big mistakes early on...