Attention

Intermediate

Mechanism that computes context-aware mixtures of representations; scales well and captures long-range dependencies.

Full Definition

Mechanism that computes context-aware mixtures of representations; scales well and captures long-range dependencies.

Keywords

query-key-value

Domains

Transformers & LLMs

Related Terms

Transformer used in

Architecture based on self-attention and feedforward layers; foundation of modern LLMs and many multimodal models.

 Self-Attention narrower than

Attention where queries/keys/values come from the same sequence, enabling token-to-token interactions.

 Self-Attention related to

Attention where queries/keys/values come from the same sequence, enabling token-to-token interactions.

 Vocabulary related to

The set of tokens a model can represent; impacts efficiency, multilinguality, and handling of rare strings.

 Context Window related to

Maximum number of tokens the model can attend to in one forward pass; constrains long-document reasoning.

 Throughput related to

How many requests or tokens can be processed per unit time; affects scalability and cost.

 Transformer related to

Architecture based on self-attention and feedforward layers; foundation of modern LLMs and many multimodal models.

Concept Map

See how Attention connects to other concepts.

Open Knowledge Graph