How to Evaluate AI Models: A Step-by-Step Checklist for Visual AI Teams

•
10 min read
Author
black and white photo of Jesse Mostipak
Director of Growth
Jesse Mostipak is the Director of Growth at Voxel51, where the work is helping humans find and trust what the brand knows, and teaching the Google knowledge graph and the LLMs answering on their behalf to do the same. That question, how knowledge gets built inside a system, is one Jesse has been chasing for years. Earlier versions of it ran through a New York City high school science classroom, data science and machine learning, and developer relations at Kaggle, Posit (formerly RStudio), and Baseten. The answer doesn't change much depending on whether the learner is a teenager, a software engineer, or a knowledge graph. Jesse holds a Master's in Education from CUNY Hunter College. LinkedIn
See all articles by Jesse Mostipak

Try Voxel51 for yourself

Updated November 5, 2026. Originally published December 17, 2024, and rewritten as a step-by-step checklist.
Evaluating an AI model means answering two questions before it ships: how good is it, and where does it fail? A single accuracy number answers the first one badly and the second one not at all. This checklist walks through how to evaluate AI models in the order the work actually happens, from building a test set you can trust to reviewing the samples a model gets wrong and deciding what to fix.
Most of these steps apply to any model that makes predictions. The examples come from images, video, 3D, and robot data, because that's where our readers work and because sequence data makes the test set step harder than it looks. Large language models (LLMs) need different metrics and a few extra steps, which this post doesn't cover. For the concepts behind each step, the model evaluation guide goes deeper.

How to evaluate AI models: the 10-step checklist

  1. Define what good means for this model.
  2. Build a test set you can trust.
  3. Choose metrics that fit the task.
  4. Score the model, then slice the results.
  5. Review the failing samples.
  6. Compare models fairly.
  7. Check confidence scores and thresholds.
  8. For robot policies, evaluate rollouts, not just predictions.
  9. Check performance across groups and conditions.
  10. Close the loop and keep evaluating after launch.

Step 1: Define what good means for this model

Start with the decision the model will support, not the metric. A defect detector that misses a crack costs more than one that flags a clean part, so recall matters more than precision. A security camera that alerts on every shadow trains people to ignore it, so precision matters more. Write down four things before you evaluate anything:
  • The task and its output: a class, a box, a mask, a track, or an action.
  • The cost of each error type: what a false positive and a false negative each cost in production.
  • The conditions that matter: lighting, weather, camera, location, object size, robot embodiment, or anything else that changes how the model sees the world.
  • The constraints it has to run under: latency on the target hardware, model size, or a false alarm budget per day. A model that scores well and misses the frame budget doesn't ship.
That list becomes your metric choice in step 3 and your slices in step 4.

Step 2: Build a test set you can trust

Most evaluation mistakes happen before a model is scored. Four checks catch the common ones.
Keep whole recordings on one side of the split. In visual and physical AI, data comes in sequences: frames from one video, one drive, or one robot episode. Neighboring frames are close to identical, so a random frame-level split puts near-identical frames in both train and test, and the score goes up for the wrong reason. Assign each video, drive, episode, scene, or location to train or test as a whole. Research on CIFAR found that 3.3% of CIFAR-10 and 10% of CIFAR-100 test images had near-duplicates in the training set, and that's a dataset of single images.
Remove near-duplicates across the split. Even a grouped split can leak when the same scene was captured twice. Near-duplicate detection over embeddings flags the repeats that metadata alone won't reveal.
Check the test labels. A wrong label in the test set punishes a correct prediction. An analysis of 10 widely used benchmarks found an average of at least 3.3% label errors in their test sets. Review a sample of test labels before you trust a score built on them.
Keep the test set sealed. Pick architectures, tune hyperparameters, and set thresholds on a validation split. Score the test set once per candidate, at the end. Every time a decision is made on test numbers, the test set turns into training data by another route, and the final score drifts upward.
Frame-level splits leak near-identical frames into the test set. Split by episode, sequence, or scene instead.

Train-test splits and cross-validation

A single train-test split, often 80/20, is the fast starting point. It's sensitive to which samples land where, especially on small or imbalanced datasets. K-fold cross-validation rotates through several splits so every sample is tested once, which gives a steadier estimate at the cost of training several times.
For sequence data, use grouped cross-validation so each video or episode stays in one fold. In scikit-learn, that's GroupKFold:
from sklearn.model_selection import GroupKFold, cross_val_score

# model: any scikit-learn classifier, X: features, y: labels
# groups has one entry per sample: the episode, video, or drive it came from
cv = GroupKFold(n_splits=5)
scores = cross_val_score(model, X, y, groups=groups, cv=cv, scoring="f1_macro")
print(f"F1: {scores.mean():.3f} +/- {scores.std():.3f}")
Every frame from an episode lands in one fold, so no fold is scored on near-copies of its own training data. A shuffled KFold would spread each episode across all five folds and lift every fold's score. If classes are imbalanced, StratifiedGroupKFold keeps the group rule and balances classes too. For detectors and policies, most teams can't afford five training runs, so the same group logic goes into a single split, and the spread comes from retraining with different seeds instead, which is the noise floor in step 6.

Step 3: Choose metrics that fit the task

Accuracy alone misleads whenever classes are imbalanced. A model that predicts "no defect" on every image scores 99% accuracy on a dataset where 1% of parts are defective, and it finds nothing. Match the metric to the task and the error costs from step 1.
TaskStart withWatch for
ClassificationPrecision, recall, F1 score, confusion matrixAccuracy on imbalanced classes
Object detectionIntersection over union (IoU), mean average precision (mAP)Which IoU threshold the mAP uses
SegmentationMean IoU, Dice coefficientSmall classes that barely move the average
RegressionMean absolute error, root mean squared errorA single average that hides a few large misses
Rare positivesPrecision-recall curveA receiver operating characteristic (ROC) curve that looks strong on imbalanced data
Robot policiesSuccess rate, with a confidence intervalSmall rollout counts
Domain-specificA custom metric that matches the jobPer-image metrics that don't match the real decision
Where to start with model evaluation metrics by task, and what to watch for with each.
The last row matters more than it looks. IBAK, which builds pipe inspection systems, computes IoU and mAP per image but also built a report-level metric that asks whether the model found every defect along the full length of the pipe, because that's the decision the inspection supports. The guide's section on which model evaluation metrics fit which task covers the tradeoffs, including how COCO-style mAP averages over IoU thresholds.

Step 4: Score the model, then slice the results

Run the evaluation on the full test set to get the headline numbers, then break those numbers down by class and by the conditions you listed in step 1. A data slice is a subset defined by one condition, such as night scenes, one camera, or small objects. Scored separately, slices show what the average hides: a model with strong overall mAP can still fail on exactly the rare conditions that cause incidents.
Three habits make slicing useful:
  • Score a baseline first. The majority-class guess, the current production model, or last quarter's checkpoint. A number means nothing until there's something next to it.
  • Slice by what matters in production, not by whatever metadata is handy. If glare causes most field failures, glare is a slice.
  • Treat small slices with care. Thirty samples can't separate 85% from 92%. Report the count next to every slice score, and don't call a gap a failure until the slice is big enough to show it.
In Voxel51, the Model Evaluation panel puts the classification report, confusion matrix, and precision-recall curve for a run in one place, and its Scenario Analysis tab scores each model on the slices you define.
Scenario analysis compares two models category by category, so a model that wins overall can be checked slice by slice.

Step 5: Review the failing samples

A slice shows where performance drops. The individual samples explain it. Open the false positives and false negatives in your weakest slice and ask three questions of each one:
  • Is the label right? If the model is right and the ground truth is wrong, that's label noise. Fix the label and the guideline that produced it, and expect the score to move on its own.
  • Has the model seen this condition before? If the condition is rare in training, it's a coverage gap: find similar samples, annotate them, and retrain. If it's absent from training entirely, it's domain shift, and you need new data, collected or generated.
  • Does it fail on well-covered conditions too? Then the problem is the model, and more data won't fix it. Change the architecture, capacity, or training recipe.
Tally the answers. The bucket with the most samples is the next thing to fix, and the guide's failures-to-data-fixes section covers each fix in more depth.
Skipping this step is how teams end up labeling more of everything when the real problem was one mislabeled class or one missing condition. In Voxel51, clicking a false positive or false negative count in the Model Evaluation panel filters the grid to the samples behind it, so you can review predictions against ground truth on the images themselves.

Step 6: Compare models fairly

When you're choosing between two models, or deciding whether a new version is safe to ship, the comparison is only as fair as its setup.
  • Freeze the test set and the metric settings. Same samples, same IoU threshold, same confidence cutoff, for every model. Version the test set and record the version next to every score, or numbers from March and November can't be compared.
  • Measure your noise floor. Retrain the same configuration with three seeds. The spread between those runs is the smallest difference that can mean anything. Two models closer than that are tied.
  • Compare per slice, and open the samples where the two models disagree. The disagreements are where the real difference lives.
Public benchmark numbers need one more caveat. When researchers rebuilt the CIFAR-10 and ImageNet test sets by following the original collection process, accuracy fell across a wide range of models, by 3 to 15 points on CIFAR-10 and 11 to 14 on ImageNet, even though the ranking between models mostly held. A benchmark score says how a model ranks on that benchmark. It doesn't say what the model will score on your data, so test on samples that look like production. In Voxel51, you can compare two or more evaluation runs side by side.

Step 7: Check confidence scores and thresholds

Production systems act on a confidence score threshold, so check first whether the scores deserve the trust. Model calibration measures how closely confidence matches how often the model is right, and modern classification networks tend to be overconfident.
  • Choose the threshold on a validation set, not the test set.
  • Choose it by the error costs from step 1, for example the threshold that meets your recall floor with the fewest false positives.
  • Re-check it after every retrain. A threshold tuned on one model version can move a long way on the next.

Step 8: For robot policies, evaluate rollouts, not just predictions

A robot policy is judged by whether the robot completes the task, so the core metric is success rate across rollouts, in simulation or on hardware. Three practices keep that number honest:
  • Report the rollout count and a confidence interval. A policy that succeeds in 90 of 100 rollouts, a realistic lab count, has an exact 95% interval of about 82% to 95%. With only 10 rollouts, 9 successes is consistent with anything from about 56% to 99.7%.
  • State the conditions and the success criteria. Initial positions, objects, lighting, and what counts as success all change the number.
  • Test under perturbation. Change object positions, the initial state, or the instruction wording. Models that score above 90% on the standard LIBERO benchmark collapsed to near zero when researchers perturbed the setup.
Then review the failed rollouts the same way you'd review failing samples in step 5. The model evaluation guide covers how to evaluate a robot policy in more depth, and the sim-to-real gap post walks through diagnosing policy failures on hardware.

Step 9: Check performance across groups and conditions

Step 4 asks how the model scores on each slice. This step asks whether the test set can answer that question for the groups that matter. A model that performs well overall can still fail for a specific group of people or a specific environment, and a test set that barely includes that group won't show it.
  • Look for imbalance in the data first. If the test set under-represents a group or condition, the score for that group is both noisy and easy to ignore.
  • Stratify within your grouped split. Keep episodes together (step 2) and balance groups across train and test inside that rule. Proportional representation isn't always enough: a group that's 2% of the data gets a 2% share of the test set and a score with error bars too wide to act on. For the groups you must get right, add test samples beyond their natural share.
  • Score each group separately, using the same slicing habit from step 4. For models that analyze people, that can mean skin tone, age, or clothing. For a driving model, it can mean region, road type, or weather.
The goal isn't one fairness score. It's knowing, before launch, which groups or conditions the model serves worst.

Step 10: Close the loop and keep evaluating after launch

Evaluation doesn't end at launch. Production data drifts, new edge cases appear, and every retrain can regress a slice that used to work.
  • Turn failures into data. Each failure mode from steps 5 and 8 is a target: find similar samples, annotate them, and add them to training. In Voxel51, you can curate training data from model failures and send selected samples to annotation.
  • Keep a regression set. Save the samples that broke earlier models, version the set like the test set in step 6, and re-score every new version on it.
  • Monitor in production. Log predictions and watch for shifts in input data, confidence, and error rates.
  • Use A/B tests carefully. Comparing two model versions on live traffic shows which one serves users better, as long as you also check the slices from step 4 and not just the overall result.
  • Re-evaluate on fresh data. A test set built a year ago may no longer look like production.

How to run this checklist in Voxel51

The scoring, review, and comparison parts of steps 4 through 6 can be run in a browser without writing code. In the free tier of Voxel51:
  1. Add the VisA Anomaly Detection demo: 2,162 industrial inspection images with evaluated predictions from two models, PaDiM and PatchCore. Demos count toward the three free dataset slots.
  2. Step 4: open the Model Evaluation panel for either run and read the classification report, confusion matrix, and precision-recall curve, then open the Scenario Analysis tab to score both models across the 12 object categories.
  3. Step 5: click a false positive or false negative count to filter the grid to those samples, with predictions drawn over ground truth.
  4. Step 6: compare the two runs side by side, click a false positive or false negative count for either run, then tag those samples and save them as a subset to curate or relabel.
It takes a work or .edu email to sign up. Start with the VisA Anomaly Detection demo, and Try Voxel51 free lists what the free tier includes.

Frequently asked questions

Evaluating a visual or physical AI model? Try Voxel51 for free · Read the model evaluation guide · Book a demo
black and white photo of Jesse Mostipak
Director of Growth
Jesse Mostipak is the Director of Growth at Voxel51, where the work is helping humans find and trust what the brand knows, and teaching the Google knowledge graph and the LLMs answering on their behalf to do the same. That question, how knowledge gets built inside a system, is one Jesse has been chasing for years. Earlier versions of it ran through a New York City high school science classroom, data science and machine learning, and developer relations at Kaggle, Posit (formerly RStudio), and Baseten. The answer doesn't change much depending on whether the learner is a teenager, a software engineer, or a knowledge graph. Jesse holds a Master's in Education from CUNY Hunter College. LinkedIn
See all articles by Jesse Mostipak

Try Voxel51 for yourself

Related posts

Boost your visual and physical AI workflows.
View all
by Jesse Mostipak, Marketing
Physical AI
•
by Harpreet Sahota, DevRel
Model evaluation, Product and news
•