Learning Rate Schedule

Gradually increasing learning rate at training start to avoid divergence.

Limiting gradient magnitude to prevent exploding gradients.

Optimization using curvature information; often expensive at scale.

Matrix of second derivatives describing local curvature of loss.

Variability introduced by minibatch sampling during SGD.

The shape of the loss function over parameter space.

A narrow minimum often associated with poorer generalization.

A wide basin often correlated with better generalization.