Modern autonomous vehicles collect staggering amounts of data. Hour after hour, they record the ordinary rhythm of driving: following traffic, waiting at red lights, staying centered in the lane. The moments that actually matter for safety occupy only a tiny fraction of that footage. Collecting terabytes of data is easy. Finding the handful of frames that matter is the hard part.
The imbalance isn’t hypothetical. In the SearchAD benchmark, researchers at Mercedes-Benz and Esslingen University of Applied Sciences assembled 423,798 frames from 11 established driving datasets to study rare image retrieval. Within that massive collection, the 30 rarest classes each appear in fewer than 250 frames, and some appear fewer than 50 times.
Cars are everywhere. Every AV team knows to look for emergency responders. No one plans for donkeys, and that’s the point: the long tail is full of rare objects and situations you cannot anticipate in advance. These rare cases are where a perception failure becomes a safety failure.
The challenge, then, is not collecting more data. It’s discovering the rare, safety-critical examples already buried inside the data you have.
Key takeaways
The hardest problem in autonomous driving is not collecting more data, it is discovering the rare, safety-critical cases already buried in the data you have. As datasets grow, the bottleneck shifts from acquisition to discovery, and more data collection makes the search problem worse, not better.
Autonomous driving datasets are heavily imbalanced toward a handful of common classes. Cars and pedestrians account for the overwhelming majority of annotations, while safety-critical classes such as emergency responders appear in a few dozen frames or fewer.
The long tail contains two kinds of rarity: rare classes and rare scenarios. Rare classes are uncommon objects like donkeys or ambulances. Rare scenarios are common objects arranged in uncommon ways, like a cyclist carrying a ladder, and they are harder to find because nothing about the individual objects is unusual.
Standard tools miss rare cases in distinct ways. Label-based search can only return predefined classes, aggregate metrics like mAP can hide rare-case regressions and the long tail is open-ended, so no list of edge cases is ever complete.
Similarity search using image embeddings can surface rare cases from a single example, no predefined label required. One example, visual similarity and human review can outperform manually scrolling through hundreds of thousands of frames, and the workflow improves as embedding models do.
Why are AV datasets so imbalanced?
A handful of common object classes account for the overwhelming majority of annotations in autonomous driving datasets.
Consider nuScenes, one of the most widely used public datasets for autonomous driving. Its 1.4 million 3D bounding boxes span 23 object classes, but those annotations are anything but evenly distributed. Cars and adult pedestrians dominate the dataset. Ambulances, strollers and wheelchairs sit near the bottom of the distribution.
nuScenes in FiftyOne. Across the trainval split's 23 classes, cars (493,322 annotations) and adult pedestrians dominate while safety-critical rarities like strollers (1,072) and ambulances (49) barely register. Data: nuScenes (Caesar et al.).
We loaded the nuScenes trainval split into FiftyOne and counted: 493,322 car annotations, 1,072 strollers and 49 ambulances. For every stroller the dataset can teach a model about, it offers roughly 460 cars.
The imbalance matters because perception systems learn from examples. By the time a model has seen hundreds of thousands of cars, another ordinary sedan contributes relatively little new information. A stroller is different. Those 1,072 annotations have to teach the model what a stroller looks like across viewing angles, distances, lighting conditions and levels of occlusion.
This imbalance shapes everything that follows. Most of the data reinforces situations the model has already learned to recognize. As datasets grow, the scarce resource is no longer data. It’s the small fraction of examples that can still teach the system something new.
But frequency is only part of the story. The long tail is not just about how rare its examples are. It’s also about how many different kinds of rarity it contains.
What actually lives in the long tail?
The long tail is not a short list of unusual examples. It is a large, disordered region of uncommon objects and uncommon situations.
Plot the class frequencies of any large driving dataset and the shape is almost always the same: a handful of towering bars on the left, followed by a long, shallow slope stretching to the right.
A long tail, even among rare classes. SearchAD's classes fall from roughly 125,000 traffic cones to single-digit donkeys and medical personnel, and the safety-critical categories sit deepest of all. Data: SearchAD (Embacher et al., 2026).
In the SearchAD dataset, that slope contains 90 rare categories and more than 513,000 annotations. The tail is not a rounding error at the edge of the data, it’s substantial territory in its own right.
It’s also disorganized territory.
Rare cases are scattered, not filed. Each SearchAD class is spread across 11 source datasets (nuScenes, Mapillary, BDD100K, and more), and the rarest surface in just one or two, with no clean structure to search. Data: SearchAD (Embacher et al., 2026).
Rare examples do not cluster in one convenient corner of the dataset. They are scattered across source datasets, geographies, weather conditions and times of day. The challenge is not simply that the tail is large. It’s that there is no obvious structure to explore. There is no single folder of edge cases waiting to be opened and inspected.
The long tail contains two fundamentally different kinds of rarity:
Rare scenarios are common objects arranged in uncommon ways. A pedestrian emerging from behind a truck. A person directing traffic after an accident. A cyclist carrying a ladder. A person using walking sticks to cross the street. Nothing about the individual objects is unusual. A truck is common. A pedestrian is common. What makes the scene rare is the interaction between them: the combination, context or behavior.
One is defined by what appears. The other is defined by how it appears.
SearchAD contains both kinds, and both hide in plain sight. Open one of its donkey frames at full size and nothing immediately stands out. An ordinary road. An ordinary day.
Nothing stands out. Two donkeys graze at the edge of an ordinary road. Across SearchAD's 344,966 annotated frames, donkeys appear in just 9. Image: Mapillary Traffic Sign Dataset via SearchAD (Embacher et al., 2026).
You could scroll past it without a second glance, yet across 344,966 annotated frames, annotators found donkeys in just 9 of them.
The needle, revealed. Zoom into those two specks beside the road and they resolve into a pair of donkeys, each a labeled box. SearchAD records just 14 donkey annotations across 9 frames. Image: Mapillary Traffic Sign Dataset via SearchAD (Embacher et al., 2026).
The same is true of pedestrians using walking sticks. At first glance, they look like ordinary pedestrians. Collectively, they are one of the rarest and most safety-critical categories in the dataset.
Ordinary at a glance. A FiftyOne patch grid of walking-stick pedestrians, one of the rarest and most safety-critical categories in SearchAD. Data: SearchAD (Embacher et al., 2026).
The long tail is not simply a collection of rare objects. It is an open-ended space of rare classes, rare scenarios and novel combinations that cannot be exhaustively listed in advance.
If rare examples are the problem, why not simply collect more data?
Why can't you just collect more data?
Collecting more driving data mostly produces more of what you already have: ordinary driving.
Early in development, volume genuinely helps. A model that is still learning what intersections, merges and crosswalks look like benefits from nearly every mile. But the returns invert as the model matures. Once common cases are handled, each additional hour of footage mostly reinforces patterns the model has already learned.
The reason is simple: scale does not change the shape of the distribution. The data is skewed because the world is skewed, so uniform collection reproduces the same imbalance at any size. If a rare class appears only a handful of times in a large dataset, doubling the dataset may add a few more examples while adding vastly more of the common classes.
Nor can you simply create the missing cases on demand. Many rare cases are dangerous or impossible to stage. No one is going to choreograph a pedestrian stepping out from behind a truck. And even the cases you could stage would not solve the problem, because the tail is open-ended. Staging a donkey solves one example, but the space of possible edge cases has no end.
There’s a final complication: collecting more data can make discovery harder. Every additional hour of footage increases the volume that rare cases are buried within. Past a certain point, more collection delivers diminishing returns on common cases while increasing the effort required to find the rare ones. The bottleneck has shifted from acquisition to discovery.
The SearchAD researchers describe it the same way: "the key challenge shifts from collecting more data to efficiently identifying the most relevant samples."
But discovery introduces a different set of problems, ones that go well beyond volume.
Why is the long tail so hard to fix?
The long tail exposes a deeper problem: our tools were built to manage known categories, not discover unknown ones. Search, labels, metrics and prioritization each have blind spots when it comes to rare cases, and solving one problem only reveals the next.
First, you have to find the rare cases.
You cannot manually inspect hundreds of thousands of frames. When the SearchAD team exhaustively annotated their 423,798 images, it took a specialized annotation company eight months, with annotators spending more than a minute per image and every image reviewed at least twice. That is the cost of finding rare cases by hand, and most teams cannot afford it.
One filter, many datasets. Filtering a rare class in FiftyOne pulls every instance into a single view, here spanning KITTI and Mapillary, with no per-dataset folder digging. Data: SearchAD (Embacher et al., 2026).
Automated retrieval helps, but it is far from solved. Even after fine-tuning the best available models on SearchAD's own benchmark, the researchers described absolute retrieval performance as "unsatisfactory." The data is there, but locating it is its own challenge.
Then, you have to describe cases no one anticipated.
Suppose you could search efficiently. You still need words for what you are searching for, and the long tail grows faster than any labeling ontology. Traditional datasets depend on predefined classes, so if nobody anticipated "person wearing a trash bag in the rain" or "wheelchair tipped over," those situations cannot be retrieved through label-based search.
This is what we’d call ontological rigidity: the vocabulary of the dataset was fixed before the world finished surprising it. Even SearchAD, with 90 carefully chosen rare categories, is still a fixed list. Many important scenarios are invisible simply because no one defined them in advance.
Then, you have to measure what mAP hides.
Suppose you found and labeled them. Your metrics still will not tell you how you are doing. Aggregate metrics such as mean average precision (mAP) reflect the distribution of the dataset, and because common classes dominate the evaluation set, meaningful improvements or regressions on long-tail cases can disappear inside an otherwise strong overall score.
Aggregate mAP hides the long tail. Evaluated in FiftyOne, the detector with the higher overall mAP (0.396 vs 0.381) wins the common classes yet trails on rare class after rare class. Data: FiftyOne quickstart.
A model can post better mAP while getting worse at the rare cases that matter most. Improving the headline number does not necessarily improve visibility into the long tail.
Then, you have to prioritize by importance, not frequency.
Even perfect measurement leaves a judgment call, and frequency-weighted thinking gets it exactly backward. Missing another sedan changes very little. Missing the only pedestrian using crutches could represent a serious safety blind spot. The examples that matter most are often the ones that appear least.
And the long tail is never finished.
New weather conditions, new behaviors, new vehicle types, new combinations of familiar objects: every deployment introduces situations that did not previously exist. The tail is open-ended by nature, so there is no finish line where the list of rare cases is complete.
You cannot collect, label or benchmark your way through the long tail. You need a way to surface it.
How do you find rare cases without labels?
To search the long tail, you have to stop relying on labels alone.
Traditional search asks: What labels did someone assign this image? That question can only return what an ontology anticipated, and the previous section showed why that fails for the long tail. The person wearing a trash bag in the rain never received a label, so no label query will ever find them.
Similarity search asks a different question: What else looks like this? Instead of matching label strings, it works from the content of the images themselves. It uses embeddings that capture patterns in the images, placing visually and conceptually similar frames near one another. That distinction matters because rare scenarios escape fixed ontologies while still sharing meaningful visual structure. Nobody defined a class for the trash bag pedestrian, but frames like it still resemble each other.
You can see the difference in an embeddings view, where an entire dataset is laid out by what its frames actually contain rather than what they were named.
Find a single cyclist, and the neighborhood around it surfaces the rest: no cyclist class required, no manual inspection of hundreds of thousands of irrelevant frames, one example doing the work of a query. This is the same principle tested in SearchAD's image-to-image retrieval protocol, where a handful of example frames serves as a query instead of a predefined class.
To be clear, retrieval models are not a solved problem, as SearchAD's own benchmark shows. But the workflow does not require perfect retrieval. One example, visual similarity and human review can dramatically outperform manually scrolling through 400,000 frames, and the process improves as the underlying models do.
The long tail is the large set of rare objects, scenarios and combinations that appear infrequently in driving data, such as animals on the road, emergency responders or pedestrians using mobility aids. These cases are individually rare but collectively vast, and they are where perception models are most likely to fail.
Uniform collection reproduces the world's own skew. Doubling a dataset adds vastly more common examples and only a few rare ones, while burying the rare cases deeper. In the SearchAD benchmark, the 30 rarest classes each appear in fewer than 250 of 423,798 frames.
A rare class is an uncommon object, such as a donkey or an ambulance. A rare scenario is common objects arranged in uncommon ways, such as a pedestrian emerging from behind a truck. Rare scenarios are harder to find because nothing about the individual objects is unusual.
Rare scenario failures hide inside their parent class's score, so a model can improve on pedestrians overall while getting worse at pedestrians using crutches. Rare classes also contribute so few evaluation examples that their scores are noisy, and benchmark protocols often merge or drop them entirely.
Similarity search. Image embeddings place visually similar frames near one another, so a single example of a rare case can retrieve others like it, with no predefined class and no manual review of hundreds of thousands of frames.
SearchAD is a large-scale rare image retrieval benchmark built by researchers at Mercedes-Benz and Esslingen University of Applied Sciences. It contains 423,798 frames from 11 established driving datasets, with 513,265 manual annotations across 90 rare categories, and tests how well retrieval methods locate rare, safety-critical cases.
