Context Compression

Attention mechanisms that reduce quadratic complexity.

Routes inputs to subsets of parameters for scalable capacity.

Chooses which experts process each token.

Capabilities that appear only beyond certain model sizes.

Allows model to attend to information from different subspaces simultaneously.

Stores past attention states to speed up autoregressive decoding.

Encodes positional information via rotation in embedding space.

Encodes token position explicitly, often via sinusoids.