Autonomous driving data curation

Autonomous driving data curation is the practice of deciding which slices of a vehicle fleet's recorded data are worth keeping, labeling, and training on. An autonomous vehicle (AV) fleet generates petabytes of camera, LiDAR, and radar logs, and the overwhelming majority of it is uneventful highway driving the models already handle. Curation is how teams find the needles in that haystack: the rare events, edge cases, and underrepresented scenarios that actually move model performance.

The math makes curation unavoidable. Labeling costs scale with data volume, but model improvement scales with data diversity and difficulty. A million nearly identical frames of clear-weather lane keeping teach a model less than a hundred examples of an occluded pedestrian at dusk. Curation is the discipline of spending the labeling and training budget where it counts.

Curation typically combines multiple signals over the same fleet logs. Trigger-based mining flags moments around disengagements, hard braking, or model disagreement. Embedding and semantic search lets engineers query the logs for scenarios by content, like construction zones at night or cyclists in rain, without exhaustive metadata tags. Coverage analysis maps the dataset against a taxonomy of scenarios to reveal gaps, and deduplication in embedding space collapses thousands of near-identical frames into representative samples. The output is a curated subset, orders of magnitude smaller than the raw logs, that flows into annotation, training, and scenario-based evaluation. In a mature setup, this process runs continuously as one stage of a data engine.

The safety case for autonomous driving lives in the long tail. Common scenarios are largely solved, and the remaining failures cluster in situations too rare to appear reliably in any uncurated sample of driving data. Curation matters because it is the only economically viable way to concentrate training and evaluation on those situations at petabyte scale. It also compounds: teams that curate well build datasets whose difficulty and diversity grow with every fleet mile, while teams that sample randomly pay ever more to relearn what their models already know. ML engineers building perception and planning systems feel this daily, and the leaders funding labeling budgets should care just as much.

The principles are shared, but the scale and structure are different. AV curation operates on multimodal, time-synchronized sensor streams measured in petabytes, where selection happens at the level of driving scenarios rather than individual files. It also carries a safety dimension, since missing a class of rare events in the dataset can translate directly into on-road failure modes.

Anything the current system handles poorly or has rarely seen. Typical examples include pedestrians emerging from occlusion, unprotected turns in dense traffic, emergency vehicles, debris on the road, unusual vehicle types, and severe weather. The definition shifts over time, since yesterday's rare event becomes well-covered after a few curation cycles, and mining moves further down the long tail.

Data Curation, Data engine, Longtail investigation, Data slice, Embedding and semantic search, Autonomous driving annotation, Sensor fusion, Distribution shift / domain shift.