What is model calibration?
Model calibration measures whether a model's stated confidence lines up with how often it is actually right. If predictions made with 0.9 confidence are correct roughly 90 percent of the time across the dataset, the model is well calibrated. If they are correct only 60 percent of the time, the model is overconfident and its scores overstate certainty.
Calibration is separate from accuracy: a model can be accurate yet poorly calibrated, or calibrated yet inaccurate.
Key takeaways
- Calibration compares confidence scores to observed accuracy.
- Many modern models are overconfident and need recalibration.
- It is distinct from accuracy and must be measured separately.
How it works
Calibration is assessed by grouping predictions into confidence bins and comparing the average confidence in each bin to the actual accuracy in that bin, often visualized as a reliability diagram and summarized by expected calibration error. Techniques such as temperature scaling, Platt scaling, or isotonic regression adjust the scores after training to bring confidence in line with accuracy.
Why it matters
Any decision that relies on a confidence threshold, from filtering predictions to routing uncertain cases for human review, assumes the scores mean something. Poor calibration quietly breaks that assumption and leads to thresholds that behave differently than expected. Calibrated confidence is essential wherever a model's certainty drives an action.
Frequently asked questions
Why are neural networks often overconfident?
Modern networks tend to push scores toward 0 or 1 during training, so their confidence often exceeds their true accuracy without recalibration.
How do you measure calibration?
Commonly with a reliability diagram and a summary like expected calibration error, which compares predicted confidence to observed accuracy across bins.
Related terms