Gradient Clipping

Optimization using curvature information; often expensive at scale.

Matrix of second derivatives describing local curvature of loss.

Neural networks can approximate any continuous function under certain conditions.

Allows gradients to bypass layers, enabling very deep networks.

A narrow minimum often associated with poorer generalization.

A wide basin often correlated with better generalization.

Adjusting learning rate over training to improve convergence.

Gradually increasing learning rate at training start to avoid divergence.