What is knowledge distillation?
Knowledge distillation transfers what a large, accurate teacher model knows into a smaller, faster student model. Rather than training the student only on hard labels, you also train it to match the teacher's full output distribution, which carries richer information about how the teacher sees each example. The student ends up much smaller while retaining a large share of the teacher's accuracy.
It is one of the main tools for making capable models cheap enough to run in production or on constrained hardware.
Key takeaways
- Distillation trains a small student to imitate a large teacher.
- The student learns from the teacher's soft outputs, not just labels.
- It compresses models while preserving much of their accuracy.
How it works
The teacher generates outputs, often softened probability distributions, on training data. The student is trained to match those outputs, sometimes alongside the true labels. Matching the teacher's distribution exposes the student to relationships between classes that hard labels hide, which is what lets a much smaller model perform well.
Why it matters
Large models are accurate but expensive to serve, and distillation is a practical bridge to fast, low-cost inference without starting over. Combined with quantization, it is a standard step in shrinking models for real-time or edge deployment, which matters for visual and physical AI systems running under tight latency budgets.
Frequently asked questions
What are the teacher and student models?
The teacher is a large, high-performing model, and the student is a smaller model trained to reproduce the teacher's behavior efficiently.
How is distillation different from quantization?
Distillation transfers knowledge into a smaller architecture, while quantization reduces the numerical precision of an existing model. They are often used together.
Related terms