Distillation

Simple Definition

Distillation (also called knowledge distillation) is a training process where a smaller AI model, called the “student”, learns to replicate the outputs of a larger, more powerful model, the “teacher.” The goal is to compress the knowledge of a big expensive model into a smaller, faster, cheaper one.

A Simple Analogy

Imagine a world-class expert (the teacher model) who writes detailed explanations of everything they know. A student reads all of those explanations and learns to reason similarly, without having lived the expert’s full career. The student can then answer questions almost as well as the expert, but in a fraction of the time.

How Distillation Works

A large, capable model (teacher) generates responses to thousands of inputs
A smaller model (student) is trained on those responses, learning to produce similar outputs
The student doesn’t learn from raw internet text directly. It learns from the teacher’s filtered, high-quality outputs
The result is a smaller model that has “absorbed” much of the teacher’s knowledge

Why Distillation Matters

Running frontier AI models is expensive. Distilled models let companies and developers:

Deploy capable AI at lower cost
Use AI on devices with limited computing power
Reduce response time (smaller models are faster)
Keep performance close to the original without the resource demands

Famous Examples

DeepSeek R1: distilled reasoning capabilities from a large model into smaller, highly efficient versions
Phi series (Microsoft): small models that punch above their weight class because of distillation on high-quality data
Gemma (Google): efficient models partly built using distillation techniques

Distillation vs. Quantization

Both make models smaller and faster, but differently:

Distillation: trains a new smaller model from scratch, guided by a larger one
Quantization: compresses an existing model by reducing numerical precision

Distillation produces a genuinely smaller model. Quantization produces a compressed version of the original.

Model Parameters, distilled models have fewer parameters than their teachers
SLM, many small language models are built using distillation
LLM, the teacher models in most distillation setups
Fine-Tuning, distillation is a specific form of training that shares methods with fine-tuning
Synthetic Data, teacher model outputs are a form of synthetic data used in distillation