Loss Landscape

A narrow minimum often associated with poorer generalization.

A wide basin often correlated with better generalization.

Adjusting learning rate over training to improve convergence.

Gradually increasing learning rate at training start to avoid divergence.

Optimization problems where any local minimum is global.

Optimization with multiple local minima/saddle points; typical in neural networks.

A point where gradient is zero but is neither a max nor min; common in deep nets.

Variability introduced by minibatch sampling during SGD.