Attention Head

Allows model to attend to information from different subspaces simultaneously.

Stores past attention states to speed up autoregressive decoding.

Encodes positional information via rotation in embedding space.

Encodes token position explicitly, often via sinusoids.

Built-in assumptions guiding learning efficiency and generalization.

The range of functions a model can represent.

Tradeoffs between many layers vs many neurons per layer.

Prevents attention to future tokens during training/inference.