Scenario mining

Scenario mining is the work of digging through mountains of recorded data to pull out the particular situations that matter. A fleet of autonomous vehicles or robots generates an overwhelming stream of logs during normal operation, most of which is unremarkable. Scenario mining is how teams find the needles in that haystack, retrieving the moments that fit a description of interest, such as a pedestrian stepping off a curb, a vehicle cutting in during heavy rain, or an unprotected left turn at a busy intersection. The output is a curated collection of scenarios rather than a pile of raw logs.

What distinguishes scenario mining from general data curation is its focus on situations that unfold over time, defined by the interaction of actors, conditions, and context. A scenario is not a single frame but a short episode with structure, and describing what makes it interesting often draws on maps, actor behavior, and environmental conditions. Increasingly, these situations can be retrieved using natural language descriptions matched against logs, timestamps, and the actors present. The result feeds directly into testing, evaluation, and decisions about what additional data to collect.

Scenario mining starts from a description of the situation of interest, which might be expressed in terms of actor behaviors, map features, and environmental conditions, or increasingly in natural language. That description is then matched against recorded logs, using perception outputs, map context, and the timing and trajectories of actors to locate matching episodes. Because raw fleet data is vast and only roughly labeled, effective mining relies on being able to search efficiently by content and structure rather than scanning everything by hand. Once matching scenarios are retrieved, they can be grouped, reviewed, and assembled into targeted datasets and test suites.

Scenario mining matters because the value of a huge dataset is locked up until you can find the specific situations that reveal how a system behaves. For anyone developing autonomous vehicles or robots, it is central to both safety and efficiency, since it surfaces the safety-critical and rare situations that average metrics miss, and it lets teams build focused datasets and regression tests around known problem cases. It is a practical answer to the reality that collecting data is easy but finding the moments that teach you something is hard.

Data curation is the broad practice of selecting, cleaning, and maintaining data of any kind. Scenario mining is a specialization focused on finding meaningful situations, defined by actors, conditions, and context over time, within large recorded datasets, which is especially important for autonomous systems.

Typically its rarity, difficulty, or safety relevance, such as near-misses, unusual maneuvers, or challenging conditions. These are the situations that expose weaknesses a model might otherwise pass over, so they are prime candidates for targeted testing and data collection.

Scenarios can be defined by actor behaviors, map features, and environmental conditions, and increasingly by natural language descriptions. These descriptions are matched against recorded logs using perception outputs, map context, and actor trajectories to retrieve the moments that fit.

Data curation, Longtail investigation, Edge case, Failure mining, Operational design domain (ODD).