Gating Network

Capabilities that appear only beyond certain model sizes.

Empirical laws linking model size, data, compute to performance.

Formal framework for sequential decision-making under uncertainty.

All possible configurations an agent may encounter.

Encodes token position explicitly, often via sinusoids.

Techniques to handle longer documents without quadratic cost.

Attention mechanisms that reduce quadratic complexity.

Routes inputs to subsets of parameters for scalable capacity.