Parameter Sharing

Built-in assumptions guiding learning efficiency and generalization.

The range of functions a model can represent.

Tradeoffs between many layers vs many neurons per layer.

Prevents attention to future tokens during training/inference.

Neural networks can approximate any continuous function under certain conditions.

Allows gradients to bypass layers, enabling very deep networks.

Early architecture using learned gates for skip connections.

A narrow hidden layer forcing compact representations.