What is a confidence score?

A confidence score is a number a model outputs alongside each prediction to indicate how sure it is, typically ranging from 0 to 1. Setting a threshold on that score converts the model's raw outputs into concrete decisions: keep predictions above the cutoff, discard those below it. Where you set the threshold trades precision against recall.

A high confidence score is not the same as a high probability of being correct unless the model is well calibrated, which is a common and important pitfall.

Key takeaways

A confidence score expresses a model's certainty in a single prediction.
A threshold on the score decides which predictions are kept.
Confidence is only trustworthy as a probability if the model is calibrated.

How it works

Most classifiers and detectors produce a score per class, often from a softmax or sigmoid layer. Applications then apply a threshold, and in detection an additional step like non-maximum suppression removes overlapping duplicates. Sweeping the threshold is exactly what generates precision-recall and ROC curves.

Why it matters

The confidence threshold is one of the most consequential knobs in a deployed model, because it sets the balance between missing real cases and raising false alarms. Treating raw confidence as a true probability without checking calibration can lead to overconfident, poorly-tuned systems, which is why calibration and threshold selection go hand in hand.

Frequently asked questions

Is a confidence score the same as a probability?

Only if the model is calibrated. Many models are overconfident, so a 0.9 score does not always mean a 90 percent chance of being correct.

How do you choose a confidence threshold?

By looking at the precision-recall trade-off and picking the operating point that matches the tolerance for false positives versus missed detections.

Jesse Mostipak is the SEO and Content Manager at Voxel51, where the work is helping humans find and trust what the brand knows, and teaching the Google knowledge graph and the LLMs answering on their behalf to do the same. That question, how knowledge gets built inside a system, is one Jesse has been chasing for years. Earlier versions of it ran through a New York City high school science classroom, data science and machine learning, and developer relations at Kaggle, Posit (formerly RStudio), and Baseten. The answer doesn't change much depending on whether the learner is a teenager, a software engineer, or a knowledge graph. Jesse holds a Master's in Education from CUNY Hunter College. LinkedIn

Confidence score

Building visual or physical AI? 
Let's talk.