For years, the dominant mindset in computer vision—especially in the automotive space—has been to label everything. Teams would collect massive automotive datasets, send them off to an annotation team, and hope that enough brute-force labeling would lead to better models.

That approach made sense when our tools were immature and our only lever was volume. But things have changed. In 2025, the idea that we must annotate all our data to build performant AI models is not only outdated—it’s counterproductive.

The “annotate everything” era is over. And it’s time we talk about why.

In the past, automotive teams invested millions of dollars and months of time in large-scale annotation campaigns. It was the norm. But it quickly became clear that the payoff didn’t justify the cost.

Most roadway footage contains little to no signage. If you randomly sample 10% of a petabyte-scale video dataset, what you’ll likely get is hours of open highway or crowded urban streets—scenarios that are already overrepresented. What you won’t get are the rare cases that actually improve model performance: speed limit signs, temporary construction warnings, edge-case intersections.

By annotating everything uniformly, teams were spending an extraordinary amount of effort to reinforce patterns the model had already mastered—while missing the data that would actually help it improve.

One of the most damaging myths in computer vision is that more labeled data automatically yields better models. This is simply not true as throwing more annotations at the problem is often a waste of money. The better question is: What data actually matters for improving perception systems performance? And how can we find that data more efficiently?

This question led us to develop a different kind of tooling—focused not on labeling more, but on identifying what to label. Today, with the help of semantically rich foundation models like OpenAI’s CLIP, we can embed entire datasets and visualize the structure of the data before touching a single label.

Using these embeddings, we can automatically select representative samples, filter for outliers, and ensure we’re building a dataset that is both diverse and targeted. Instead of blindly sampling 10% of your data, you can sample the right 10%—the subset that actually improves model accuracy.

This isn’t just theoretical. I’ve seen teams use this approach to achieve better performance with fewer labels, lower cost, and faster turnaround.

A while back, I wrote a blog post titled Annotation is dead. That headline ruffled a few feathers—but the core point holds. Annotation isn’t disappearing, but the way we think about it must change. Thanks to foundation models, we can now auto-label up to 40% of a dataset with reasonably high accuracy. With the right QA tools, we can accept, reject, or correct those labels without manual inspection of every sample.

This shift drastically reduces the time and cost of building training sets. What used to take 3–4 months and cost millions can now be done in a few hours with significantly less human effort. Recent research has demonstrated that verified auto-labeling can achieve up to 95% of human-level performance while cutting labeling costs by up to 100,000x. Models trained solely on these auto-labels often match or even exceed the performance of those trained on traditional human-labeled data, particularly for challenging edge cases where foundation models can generalize better than human annotators working at scale.

In the automotive industry, where the bar for accuracy is extraordinarily high, this shift is a game changer.

Whether you’re building perception systems for ADAS or full autonomy, you need performance at the edge, where rare events happen. By focusing annotation efforts where they matter most, you can get to that performance faster, with lower spend and more confidence in the result.

You're no longer wasting time labeling redundant data. You're strategically building better models from the start. I recently joined AI in Automotive Podcast to discuss this in detail, listen to the full episode if you’re dealing with video datasets that seem to have a mind of their own.

At Voxel51, we’ve spent the last several years building tooling that reflects this new philosophy: putting data at the center of visual AI. I believe that in the next few years, we’ll see fewer teams asking “how many annotations do we need?” and more teams asking “how much of this work can we avoid?”

This isn’t just a technical shift—it’s a strategic one. The annotate-everything era was a product of its time. But we’ve outgrown it. If we’re serious about deploying robust perception systems and real-world AI, we need smarter data pipelines—not bigger labeling budgets. The future of automotive datasets lies in intelligent selection, not exhaustive annotation.