Causal Mask

Intermediate

Prevents attention to future tokens during training/inference.

Full Definition

Prevents attention to future tokens during training/inference.

Keywords

autoregressive

Domains

AI Economics & Strategy

Related Terms

Attention Head related to

A single attention mechanism within multi-head attention.

 Multi-Head Attention related to

Allows model to attend to information from different subspaces simultaneously.

 Key-Value Cache related to

Stores past attention states to speed up autoregressive decoding.

 Rotary Positional Embeddings related to

Encodes positional information via rotation in embedding space.

 Parameter Sharing related to

Using same parameters across different parts of a model.

 Inductive Bias related to

Built-in assumptions guiding learning efficiency and generalization.

 Expressivity related to

The range of functions a model can represent.

 Depth vs Width related to

Tradeoffs between many layers vs many neurons per layer.

Concept Map

See how Causal Mask connects to other concepts.

Open Knowledge Graph