What is linear probing?
Linear probing asks a simple question of an embedding model: is the information needed for a task already laid out in its vectors? The embedding model is frozen, a single linear layer is trained on its outputs, and the probe's accuracy becomes the measure. If a mere linear boundary separates the classes, the representation has done the hard work already.
It is the standard evaluation for self-supervised and foundation models, and a fast, practical way to compare embedding models on your own data.
Key takeaways
- Linear probing trains only a linear classifier on frozen embeddings.
- Probe accuracy measures how well embeddings organize task information.
- It is a fast, standard way to compare representations.
How it works
Embeddings are computed once for a labeled dataset, then a linear classifier is fit on them, often just logistic regression. Because the encoder never updates, the result isolates representation quality from model capacity. Comparing probe accuracy across embedding models, layers, or training checkpoints reveals which representation best suits the task, at a fraction of fine-tuning's cost.
Why it matters
Teams choosing an embedding model for search or curation need evidence, not leaderboard faith, and a linear probe on in-domain data is the cheapest rigorous test. The gap between probe accuracy and full fine-tuning also indicates how much task-specific signal remains locked in the model, informing whether fine-tuning is worth it.
Frequently asked questions
Why a linear classifier specifically?
Its simplicity is the point: if a linear boundary suffices, the embedding space itself has made the classes separable, which is the property being measured.
How does linear probing differ from fine-tuning?
Probing freezes the model and trains only the final layer, measuring the representation as-is. Fine-tuning updates the model itself, measuring its potential after adaptation.
Related terms
Go deeper