Seemingly well-built AI models that score high on key performance metrics and pass all the right QA and evaluation checks can still struggle when exposed to real-world data. Once deployed, the same model that performed reliably in development begins missing objects, producing false positives, and exhibiting unstable behavior in messy, unexpected environments.
Recent incidents make this painfully clear. Tesla's self-driving cars
struggled to detect pedestrians and obstacles in low-visibility conditions. Walmart’s
anti-theft prevention systems flagged innocent customer behaviors as theft. These examples show how perceived “accurate” models fail in real-world scenarios. Even enterprises with large budgets, world-class engineering teams, and access to massive datasets aren’t immune to these failure points.
This frustrating gap between "works in development and testing" and "works in the real world" keeps executives (even in the most sophisticated AI orgs) up at night. And for good reason, the reality is that more than
80% of AI projects fail.
To address these challenges, we’re excited to announce
Scenario Analysis, a new capability in FiftyOne’s model evaluation workflow, designed specifically to uncover and solve these context-sensitive failures. Scenario Analysis uncovers hidden model weaknesses by analyzing performance across different slices of user-specified data, such as stratifying by scene types, weather, object size, or custom user-defined scenarios. This exposes
where and why a model fails and enables proactive steps to close the data gaps.
The uncomfortable truth about AI success rates
With such a high failure rate, conventional analysis would indicate investigating model architectures, algorithms, or even the tools and technology used. But what might be lurking behind those failures is oftentimes the underlying raw training data.
Data-related issues appear more frequently than many expect. In conversations with teams deploying computer vision in high-stakes environments—automotive, manufacturing, defense, healthcare—the most common pattern we hear is that model failures often stem from the data itself.
The most common data issues that lead to downstream visual model failures center around:
- Data bias: Imbalanced datasets with overrepresented or underrepresented classes cause skewed predictions, resulting in bias.
- Edge cases: When training data lacks rare or unusual scenarios, models fail to learn those conditions and fail to recognize edge cases in the real world.
- Labeling errors: Incorrect or missing annotations teach the model wrong associations.
- Low-quality samples: Blurry, low-resolution, or duplicate images confuse the model during training and inference.
Unlike model architecture bugs or training instability, which tend to surface early, failures rooted in data silently degrade performance until an incident brings them to light.
Beyond basic checks: Why preventing failures requires deeper data and model understanding
Organizations are increasingly investing in building solid foundations for their ML workflows. From data curation and annotation to model training, evaluation, testing, and QA,
these workflows reduce the manual effort and speed up development. What’s often missing, however, is a more structured and systematic approach to analyzing data and models during development.
Teams need to go beyond surface-level validation to understand their visual data more deeply. They must proactively identify subtle data anomalies, diversity gaps, and annotation inconsistencies—issues that often hide beneath standard evaluation but carry outsized downstream consequences. Equally important is equipping ML teams with tools that allow them to extract meaningful insights.
For example, a driving dataset may have a balanced distribution of cars, trucks, traffic lights, bicycles, and pedestrians. But if it lacks diversity, such as scenes with bicycles partially obscured by shadows or captured under different lighting conditions,
the model will struggle to generalize. Similarly, mislabeled objects in snowy or rainy scenes can teach the model incorrect associations that degrade performance.
To deliver AI applications that perform reliably in the real world, ML teams must continuously question and refine both their data and model assumptions. When done well, this deeper level of analysis unlocks long-tail benefits: faster iteration cycles, stronger generalization, and significantly lower risk of production failures.
A data-centric framework for identifying model failures
Models rarely fail uniformly; they fail in specific contexts. A model trained to detect cracks in bottle caps on an assembly line may perform well when cracks appear on flat, well-lit surfaces, but miss hairline cracks on curved surfaces. These kinds of context-driven failures highlight why traditional evaluation methods often fall short.
FiftyOne is the data engine for building high-quality datasets and visual AI models, used by thousands of ML teams such as Porsche, LG Electronics, Microsoft, and Berkshire Grey. Today, customers across healthcare, security, robotics, automotive, and other industries use FiftyOne data and model workflows to find issues early in development while reducing their model iteration time. Many customers are already using scenario-level analysis in their model evaluation workflows to build robust visual models:
- SafelyYou, which builds AI-powered fall detection for senior care facilities, uses FiftyOne’s model evaluation workflows to uncover and fix edge cases that impact accuracy. For example, when their system repeatedly flagged a dog statue as a fallen person creating high-confidence false positives, they were able to visually explore model outputs in FiftyOne to pinpoint the issue, retrain the model, and update their ontology to prevent future alerts. These types of insights have helped SafelyYou maintain 99% fall detection accuracy by surfacing and resolving potential failure points early.
- A Fortune 500 agriculture tech customer is using FiftyOne to evaluate and improve grain segmentation models inside harvesters, where cameras capture falling kernels. Their goal is to classify grains as husked, unhusked, or sprouting—particularly to better detect unhusked and sprouting grains. With FiftyOne’s model evaluation tools and patch-level embeddings, they discovered gaps in detecting sprouting kernels and are now mining data to strengthen classification performance across these categories.
- Ancera, a company that monitors pathogen risks in poultry operations, uses data-centric failure analysis workflows in FiftyOne to spot labeling errors in Salmonella and Coccidia samples. This process also revealed failure modes such as model bias across different chicken coop sizes. By surfacing these issues early, the team is able to curate more balanced training datasets and strengthen model generalization before deployment.
Evaluating models beyond benchmarks
Relying solely on overall performance metrics can hide nuances by averaging out important signals—a model might show 95% accuracy overall while completely failing on critical edge cases. Using FiftyOne to
investigate model evaluation results side by side across subsets of data, ML teams are able to easily and quickly spot and fix issues.
While other solutions offer very basic model evaluation capabilities and don’t provide adequate workflow flexibility, FiftyOne provides significantly more depth and flexibility to deeply analyze model behaviors across different scenarios. Going beyond aggregate metrics,
Scenario Analysis breaks down model performance across meaningful data slices (demographics, environmental conditions, object characteristics). This approach reveals where and why models are failing, not just that they're failing. Most importantly, this type of analysis becomes even more powerful when it feeds directly into active learning workflows, enabling teams to systematically address discovered gaps.
Data bias
When certain classes are overrepresented or underrepresented in a dataset, models learn imbalanced patterns. This often results in skewed predictions and bias when exposed to real-life data. Data bias often goes undetected because models can demonstrate strong overall metrics while systematically failing on certain underrepresented classes, creating fairness and reliability issues.
Failure mode:
Unbalanced classes
How FiftyOne helps find these:
- Using scenario analysis, compare how different subsets of data (e.g., lighting, weather conditions, time of day, camera angles) perform to find data gaps.
- Analyze class balance by visualizing label distributions.
- When needed, write custom plugins to analyze special or non-standard cases
Recommended fixes:
- Rebalance training data through downsampling or augmentation with FiftyOne. Collect or synthetically generate more representative samples.
Edge cases
Rare or unusual scenarios that fall outside common patterns are often underrepresented, leaving models unprepared. Hidden within otherwise healthy-looking datasets, these gaps may go unnoticed in development but
can turn into critical failure points when models face real-world conditions they weren’t trained for.
Failure mode:
Insufficient samples of certain unique conditions
How FiftyOne helps find these:
- Analyze data subsets. Visualize and explore clusters using embeddings and high-confidence false positive/negative filtering to find dense, sparse, missing, and outlier clusters.
Recommended fixes:
- Augment using synthetic data or collect and label more of those samples.
Labeling errors
Annotation quality remains one of the biggest frustrations for ML teams. Many organizations struggle with vendor-produced labels that demand extensive QA, costly, and time-consuming corrections. This creates a vicious cycle where bad labels cause model failures, and fixing them slows development. Missing or incorrect annotations weaken model learning, worsening the problem.
Recent advances in foundational model-powered
auto-labeling techniques mitigate these risks by achieving near-human performance, while cutting labeling costs by up to 100,000x.
Failure mode:
Incorrect annotation, including classification labels, bounding boxes, or segmentation masks.
How FiftyOne helps find these:
- Use Verified Auto Labeling techniques to automatically QA labels
- Visually understand the overlay of prediction vs ground_truth to spot false positives/negatives.
- Use embeddings to look for data outliers.
- Use precision-recall views and confidence scores to spot samples with annotation mistakes.
Recommended fixes:
- Correct annotation mistakes directly in FiftyOne. Trigger the relabeling, training, and evaluation pipeline.
Identify low-quality samples
Poor-quality samples, such as blurry images, poor lighting, low resolution, or duplicates, can slip through data collection pipelines and quietly reduce model accuracy by adding noise or causing overfitting. While some of these samples are essential—for example, low-light or foggy conditions in self-driving datasets to ensure models can handle adverse scenarios—imbalances in data quality can mislead training and lead models to memorize artifacts instead of learning meaningful patterns.
Failure mode:
Blurry, bright, entropy, low-resolution, exact duplicates, near duplicates
How FiftyOne helps find these:
- Data Quality workflow can easily identify these issues and enable corrective action.
Recommended fixes:
- Address problematic samples directly in the FiftyOne Data Quality panel.
Building reliable AI systems starts with data
Model failures can often be traced to mistakes made at different stages of the machine learning lifecycle, but data-related issues remain the most prevalent. The reality is simple: models are only as good as the data they learn from.
Webinar: What’s Hiding Behind Your F1-Score?
Join us on Oct 9, 2025, at @10 am PT to learn how leading ML teams are using FiftyOne data-centric workflows to uncover hidden weaknesses in their visual AI models.
High-performing visual AI systems rely on a foundation of data that has been carefully curated, deeply understood, and iteratively improved to address the unique nuances of the model's intended use. The tools and techniques teams use to build and interrogate that foundation will ultimately determine how these AI systems perform in the field.
