Warmup

Limiting gradient magnitude to prevent exploding gradients.

Optimization using curvature information; often expensive at scale.

Matrix of second derivatives describing local curvature of loss.

Neural networks can approximate any continuous function under certain conditions.

The shape of the loss function over parameter space.

A narrow minimum often associated with poorer generalization.

A wide basin often correlated with better generalization.

Adjusting learning rate over training to improve convergence.