Expressivity

Tradeoffs between many layers vs many neurons per layer.

Prevents attention to future tokens during training/inference.

A single attention mechanism within multi-head attention.

Allows model to attend to information from different subspaces simultaneously.

Early architecture using learned gates for skip connections.

A narrow hidden layer forcing compact representations.

Using same parameters across different parts of a model.

Built-in assumptions guiding learning efficiency and generalization.