Depth vs Width

Intermediate

Tradeoffs between many layers vs many neurons per layer.

Full Definition

Tradeoffs between many layers vs many neurons per layer.

Keywords

architecture tradeoff

Domains

AI Economics & Strategy

Related Terms

Causal Mask related to

Prevents attention to future tokens during training/inference.

 Attention Head related to

A single attention mechanism within multi-head attention.

 Multi-Head Attention related to

Allows model to attend to information from different subspaces simultaneously.

 Key-Value Cache related to

Stores past attention states to speed up autoregressive decoding.

 Bottleneck Layer related to

A narrow hidden layer forcing compact representations.

 Parameter Sharing related to

Using same parameters across different parts of a model.

 Inductive Bias related to

Built-in assumptions guiding learning efficiency and generalization.

 Expressivity related to

The range of functions a model can represent.

Concept Map

See how Depth vs Width connects to other concepts.

Open Knowledge Graph