Exploding Gradient

Methods to set starting weights to preserve signal/gradient scales across layers.

Techniques that stabilize and speed training by normalizing activations; LayerNorm is common in Transformers.

Randomly zeroing activations during training to reduce co-adaptation and overfitting.

An RNN variant using gates to mitigate vanishing gradients and capture longer context.

Halting training when validation performance stops improving to reduce overfitting.

Nonlinear functions enabling networks to approximate complex mappings; ReLU variants dominate modern DL.

Activation max(0, x); improves gradient flow and training speed in deep nets.

Gradients shrink through layers, slowing learning in early layers; mitigated by ReLU, residuals, normalization.