Parameter-Efficient Fine-Tuning

Intermediate

Techniques that fine-tune small additional components rather than all weights to reduce compute and storage.

Full Definition

Techniques that fine-tune small additional components rather than all weights to reduce compute and storage.

Keywords

LoRA adapters

Domains

Foundations & Theory

Related Terms

LoRA related to

PEFT method injecting trainable low-rank matrices into layers, enabling efficient fine-tuning.

 Quantization related to

Reducing numeric precision of weights/activations to speed inference and reduce memory with acceptable accuracy loss.

 Pruning related to

Removing weights or neurons to shrink models and improve efficiency; can be structured or unstructured.

 Distillation related to

Training a smaller “student” model to mimic a larger “teacher,” often improving efficiency while retaining performance.

 Reproducibility related to

Ability to replicate results given same code/data; harder in distributed training and nondeterministic ops.

 Data Lineage related to

Tracking where data came from and how it was transformed; key for debugging and compliance.

 Latency related to

Time from request to response; critical for real-time inference and UX.

 Compute related to

Hardware resources used for training/inference; constrained by memory bandwidth, FLOPs, and parallelism.

Concept Map

See how Parameter-Efficient Fine-Tuning connects to other concepts.

Open Knowledge Graph