Mean squared error (MSE) is a regression loss and evaluation metric that averages the squared differences between predicted and true values. In visual AI it scores continuous predictions like bounding box coordinates, depth maps, and keypoint locations, where lower is better.
MSE measures how far a model's numeric predictions are from the truth, on average, after squaring each error. You take the difference between each prediction and its true value, square it so positives and negatives do not cancel and large misses count more, then average across all samples. It is used both as a training loss, the thing the model minimizes, and as an evaluation metric. Because the errors are squared, MSE is reported in squared units, which is why its square root, RMSE, is often preferred for reading results in the original units.
MSE = (1/n) Σ (y_i − ŷ_i)²
Key takeaways
MSE is the average of squared errors, so larger errors are penalized disproportionately.
It is for regression, predicting continuous values, not classification.
RMSE is the square root of MSE, in the original units, and is easier to interpret.
Where it shows up in visual AI
Bounding box regression: how tightly predicted box coordinates match ground truth.
Depth estimation: per-pixel predicted distance versus true distance.
Keypoint and pose: predicted joint coordinates versus labeled ones.
Image regression and reconstruction: counts, angles, super-resolution, and denoising quality.
MSE vs related metrics
Definitions for MSE, RMSE, and MAE
Definitions for MSE, RMSE, and MAE
Metric
What it measures
Behavior
MSE
Mean of squared errors, squared units
Heavily penalizes outliers
RMSE
Square root of MSE, original units
Outlier-sensitive, easier to read
MAE
Mean of absolute errors, original units
Robust to outliers
Huber loss
Squared error for small residuals, absolute error for large ones
A tunable middle ground, robust to outliers while staying smooth near zero
Why it matters
MSE is how "how close was the prediction" becomes a single number you can optimize. Information-gain insight: the squaring is the whole story, and it cuts both ways. MSE is dominated by the worst predictions, so a model that is excellent on most samples but bad on a few is scored as if it were mediocre, and a model trained on MSE will bend itself to chase those outliers. In visual AI that is a real trap, because a handful of mislabeled keypoints or a few bad depth pixels are often label noise, not model error, and MSE will faithfully optimize toward the noise. When labels are noisy, MAE or Huber loss is usually the more honest choice.
Frequently asked questions
What is mean squared error?
The average of the squared differences between predicted and true values, used as a regression loss and metric.
What is the difference between MSE and MAE?
MSE squares the errors, so it punishes large misses harder, MAE takes absolute errors and is more robust to outliers.
Is a lower MSE always better?
Lower is better, with 0 being perfect, but MSE is scale-dependent, so it is only comparable across models on the same data and units.