Inductive Bias

The range of functions a model can represent.

Tradeoffs between many layers vs many neurons per layer.

Prevents attention to future tokens during training/inference.

A single attention mechanism within multi-head attention.

Allows gradients to bypass layers, enabling very deep networks.

Early architecture using learned gates for skip connections.

A narrow hidden layer forcing compact representations.

Using same parameters across different parts of a model.