Pruning

Training a smaller “student” model to mimic a larger “teacher,” often improving efficiency while retaining performance.

Search algorithm for generation that keeps top-k partial sequences; can improve likelihood but reduce diversity.

Stochastic generation strategies that trade determinism for diversity; key knobs include temperature and nucleus samp...

Scales logits before sampling; higher increases randomness/diversity, lower increases determinism.

Hardware resources used for training/inference; constrained by memory bandwidth, FLOPs, and parallelism.

Techniques that fine-tune small additional components rather than all weights to reduce compute and storage.

PEFT method injecting trainable low-rank matrices into layers, enabling efficient fine-tuning.

Reducing numeric precision of weights/activations to speed inference and reduce memory with acceptable accuracy loss.