Flat Minimum

Adjusting learning rate over training to improve convergence.

Gradually increasing learning rate at training start to avoid divergence.

Limiting gradient magnitude to prevent exploding gradients.

Optimization using curvature information; often expensive at scale.

A point where gradient is zero but is neither a max nor min; common in deep nets.

Variability introduced by minibatch sampling during SGD.

The shape of the loss function over parameter space.

A narrow minimum often associated with poorer generalization.