Autonomous driving reached large-scale real-world deployment earlier than most other physical AI domains, forcing teams to develop rigorous systems for failure mining, data curation, simulation, and fleet learning. It is tempting to look at general robotics and conclude that the same machinery simply needs to be rebuilt around arms, humanoids, or mobile manipulators.
There is no clean consensus on how much of the autonomous-driving playbook robotics should inherit. The reason is less philosophical than structural. Autonomous vehicle (AV) teams operate within a comparatively constrained embodiment and task structure: the system drives a vehicle through road environments using closely related control interfaces and objectives. General robotics spans manipulation, locomotion, warehouse automation, field robotics, humanoids, and dozens of other embodiments and operating environments. As a result, companies are borrowing familiar AV ideas—data flywheels, failure mining, simulation, active learning—but implementing them very differently.
Wayve emphasizes active learning and repeated fleet deployment to address the long tail problem. Physical Intelligence collects autonomous experience and recovery data from policy failures. Figure is scaling real-world deployment as a source of training data, while Google DeepMind has explored coordinated data collection across heterogeneous robot fleets. The common thread is closed-loop learning. What differs is what counts as valuable data, how it gets collected, and how easily experience transfers across tasks and embodiments.
That variation is useful. The right question is not whether robotics should learn from AV. It is which abstractions are worth stealing, and which assumptions were artifacts of driving.
Key takeaways
The AV playbook transfers at the level of data discipline, not system design. Long-tail analysis, failure mining, curation, simulation, and deployment feedback all carry over, but robotics needs different implementations.
Robotics has a harder curation problem. Data must often be normalized and selected across different tasks, sensors, action spaces, and embodiments, not just filtered for quality.
Failure mining matters more in robotics than in autonomous driving, because policies push robots into states that expert demonstrations rarely capture, making deployed failures and interventions especially valuable training signals.
Simulation transfers to robotics, but its reliable envelope is narrower than in autonomous driving. Contact-rich manipulation, deformables, and tactile sensing are far harder to simulate faithfully than vehicle dynamics.
A robotics learning loop should not be limited to robot-produced trajectories. Useful supervision also comes from teleoperation, human video, simulation, interventions, and deployed production episodes.
What actually sits underneath the AV playbook
The visible AV playbook is easy to describe: more miles, more edge cases, more simulation, more fleet feedback. The deeper lesson is different. Mature AV programs stopped treating all data as equally valuable, because driving data has a brutal longtail distribution.
nuScenes in FiftyOne. Across the trainval split's 23 classes, cars (493,322 annotations) and adult pedestrians dominate while safety-critical rarities like strollers (1,072) and ambulances (49) barely register. Data: nuScenes (Caesar et al.).
Once a model has seen enormous numbers of ordinary sedans, intersections, and lane changes, another routine frame contributes relatively little. The scarce resource becomes the small fraction of data that teaches the model something genuinely new. Rarity is also compositional: a rare scenario may contain only common objects arranged in an unusual interaction, such as a pedestrian appearing from behind a truck.
Recent datasets make that distinction increasingly explicit. KITScenes-LongTail, for example, is organized around difficult driving scenarios rather than merely uncommon object classes, and evaluates plausible behavior using multiple trajectory variants rather than assuming that one geometric trajectory fully captures the correct response.
AV teams consequently built machinery around information value: retrieve unusual scenarios, mine failures, rebalance datasets, generate counterfactual tests, and use fleet deployment to discover the next weakness. Wayve describes active learning and repeated fleet deployment as core parts of its development loop; Waymo’s Waymax exists specifically to support large-scale data-driven behavioral simulation, while newer world models target routine and rare longtail scenarios alike. That mindset transfers extremely well, yet the implementation often does not.
Longtail thinking transfers almost completely
One of the strongest ideas to import from AV is that average-case performance becomes less informative as baseline capability improves. In robotics, however, the long tail is broader than “rare objects.” It includes rare object states, contact configurations, recovery states, task transitions, human interventions, embodiment-specific failure modes, and combinations of otherwise familiar conditions. A shirt half-caught beneath another shirt is not a new semantic class. Neither is a gripper approaching a mug from a slightly bad angle. Yet either can place a manipulation policy in a state poorly represented by expert demonstrations.
This matters because closed-loop policies create their own long tail. Physical Intelligence describes the problem directly: a small positioning or grasping error moves the robot into a state that differs from the demonstration distribution; the next prediction then becomes harder, errors compound, and the episode fails. Their recent Reinforcement Learning with Experience & Corrections (Recap) work attacks precisely this problem by training on autonomous experience, including the states produced by the policy’s own mistakes.
So the AV concept transfers, but its unit of rarity changes. For driving perception, an edge case might be a donkey, an unusual emergency vehicle, or a strange pedestrian configuration. For a general-purpose robot, the important tail often lives in state × action × contact × task history. The rare event is not necessarily what the camera sees. It can be how the robot arrived there and what actions remain recoverable.
That is also why collecting indiscriminately is unlikely to be enough. DROID required 76,000 demonstrations across 564 scenes and 86 tasks to broaden manipulation diversity, while its authors explicitly describe diverse real-robot collection as expensive in hardware, labor, logistics, and safety. Google DeepMind’s AutoRT similarly used 53 robots, with up to 20 running simultaneously, to gather 77,000 real-robot episodes and deliberately used foundation models to steer collection toward more diverse or targeted settings.
The AV lesson to steal is therefore not “collect enormous amounts of data.” It is to map the tail, then spend collection budget where coverage is weak.
Robotics needs failure mining even more than AV does
If there is one AV habit robotics teams should adopt early, it is systematic failure mining. AV development organizations learned to turn model behavior into data queries. A collision risk, planner disagreement, unusual trajectory, high uncertainty, or detected scenario can become a slice that engineers retrieve from large logs, inspect, re-label, re-simulate, and eventually add to training or regression evaluation. Recent AV research continues to push scenario mining toward semantic retrieval of safety-critical and planning-relevant events rather than brute-force log inspection.
Robotics has an even stronger reason to do this because expert demonstrations systematically underrepresent the states generated by imperfect policies. Physical Intelligence’s Recap is a clean example. In addition to accumulating demonstrations, the system runs the current policy, observes its real mistakes, and uses teleoperator takeovers to demonstrate recovery from the state the robot actually reached. It then uses autonomous experience and learned value estimates to distinguish better behavior from worse behavior. Physical Intelligence reports that training on autonomous experience more than doubles throughput on some of the hardest tasks and can decrease failure rates by 2x or more.
That is the robotics equivalent of a good AV data engine, with one critical upgrade: mine not only failures, but recoveries. A failed AV episode gives engineers a high-value scenario. A failed manipulation episode can give them several distinct training assets: the precursor state, the bad action, the intervention point, the recovery trajectory, the successful continuation, and potentially a preference between alternative actions. Robotics should preserve that structure rather than collapsing the event to a binary success label.
Build dataset infrastructure that can answer questions such as show me episodes where this policy hesitated before grasping, find recoveries after object slips, or retrieve visually similar states that ended with different outcomes. Do that before the fleet becomes too large to inspect manually.
Curation discipline transfers, but robotics makes the problem harder
AV taught ML teams that dataset size and dataset quality are not interchangeable. Recent work on longtail autonomous motion planning found large gains from non-uniform sampling, with uncertainty-based curation substantially reducing collision rates relative to uniform training in its experiments. Wayve’s GAIA-2 training similarly describes a large curated dataset and balanced sampling across driving conditions rather than treating every collected frame identically.
Robotics is beginning to reach the same conclusion, only with more axes to balance. Open X-Embodiment pooled more than one million trajectories across 22 robot embodiments and found that cross-robot training can produce positive transfer, though only with sufficient model capacity, and only after converting 60 heterogeneous datasets into a common format.. For robotics, curation is not only about data quality. It also means normalizing heterogeneous data across embodiments, sensors, and action spaces—a problem AV systems face to a much smaller degree because the underlying embodiment and control space are comparatively consistent.
DataMIL's performance-aware data selection achieves the highest average performance against similarity-based baselines (AR, BR, STRAP, Flow), with the largest margins on the hardest settings.
More recent work shows why curation cannot stop at aggregation: DataMIL found that naively adding prior robot data can hurt a specialized downstream policy, while performance-aware selection from large robot datasets produced more consistent gains. Figure, which developed the Helix Vision-Language-Action (VLA) model, has published an unusually concrete version of the same lesson. In a logistics manipulation deployment, it reported that eight hours of well-curated demonstration data could produce a flexible, dexterous policy, explicitly framing the experiment around the quality-versus-quantity trade-off.
The robotics version of an AV data engine is therefore less like “find me more construction zones” and more like find me the data mixture that changes this policy’s behavior on this capability without degrading the others. That is a materially harder curation problem.
Simulation transfers selectively, but fidelity varies by task
AV organizations have invested heavily in simulation for an obvious reason: dangerous, rare driving events are expensive or unacceptable to recreate physically. According to the 2026 State of Visual and Physical AI Survey, 63% of practitioners agree that synthetic data will become the primary source of training data. Modern systems increasingly turn real logs into editable closed-loop worlds or use generative models to create controlled variations of rare scenarios. Wayve’s GAIA-4, Waymo’s latest world-model work, and Waabi’s UniSim/Copilot4D line all reflect this direction. GAIA-4 in particular closes the loop entirely, replaying real interventions with the AI Driver back in control so every model change can be measured against the same real-world scenario.
Robotics should absolutely steal the surrounding infrastructure: scenario definitions, deterministic replay, large regression suites, counterfactual testing, parallel evaluation, real-to-sim reconstruction, and explicit tracking of gaps between simulated and deployed behavior. However, it should be much more skeptical about assuming the simulator itself is equally authoritative.
Driving has difficult dynamics, but much of its environment can be represented through vehicle motion, road geometry, agent behavior, and sensor observations. Robotics introduces simulation challenges that are less central to most driving stacks: contact forces, friction, compliance, tactile sensing, deformable materials, tool interactions, articulated objects, self-occlusion, and hardware-dependent control characteristics. Recent real-to-sim work still describes reconstructing interactive robotic scenes as labor-intensive because systems must recover not only geometry but physical parameters, object states, coordinate frames, and executable interactions.
At the same time, rejecting simulation would be equally wrong. Figure reports zero-shot sim-to-real transfer for its locomotion controller using domain randomization and high-frequency feedback, while newer VLA systems are increasingly mixing synthetic and real robot data rather than choosing one exclusively.
The defensible position is therefore domain-dependent. Simulation is already highly useful where the physics can be modeled sufficiently well, as in locomotion, collision avoidance, navigation, and some rigid-body manipulation. For contact-rich and highly dexterous manipulation, treat simulation as a generator of hypotheses, coverage, pretraining data, and regression tests—not automatically as ground truth. DexSim2Real still describes sim-to-real transfer as a critical bottleneck for dexterous manipulation, spanning visual appearance, physics dynamics, and sensor characteristics. Its ablations show tactile input is worth 8.1 percent average performance on contact-rich tasks.Robotics should adopt AV’s simulation discipline while validating fidelity separately for each task and interaction regime.
Fleet learning transfers, but the fleet stops looking like a fleet
The most seductive AV lesson is the data flywheel: every deployed vehicle becomes a sensor, every deployment exposes new conditions, and the resulting data improves the next model.
Robotics companies clearly want the same thing. Figure says expanding its robot fleet generates data streams for the next generation of autonomy. Skild describes each deployment as contributing to a data flywheel. AutoRT demonstrated a centrally orchestrated collection across more than 50 robots.
But the assumption that fleet data is structurally interchangeable does not transfer as cleanly. An AV fleet usually shares a highly constrained objective—driving—and vehicles within a platform family share closely related sensors, controls, and dynamics. General robotics wants learning to move between different tasks, environments, morphologies, grippers, camera locations, degrees of freedom, and eventually entirely different embodiments. Open X-Embodiment shows that such transfer is real, but zero-shot transfer to unseen embodiments remains an active frontier rather than a default property of large-scale training. LAP notes that existing VLAs "remain tightly coupled to their training embodiments and typically require costly fine-tuning," and its own state-of-the-art result reaches roughly 50 percent average zero-shot success.
Robotics also has access to data sources that an AV-style fleet metaphor underplays. Physical Intelligence found that transfer from human video becomes substantially more effective as the diversity of robot pretraining increases, while recent work continues to study how human video, simulation, real-robot demonstrations, and autonomous experience should be mixed.
So do not build a fleet-learning system whose only ingestion path is “robot-produced trajectory.” Design the training loop around learning signals, which may include teleoperation, autonomous rollouts, interventions, failures, human video, simulation, cross-embodiment robot data, synthetic labels, and deployed production episodes. The unifying asset is not the robot itself, but the learning signal the data provides.
What robotics should actually steal
The deepest AV lesson is not a particular simulator, data format, model architecture, or labeling pipeline. It is the decision to treat autonomy development as a closed-loop data system: discover weaknesses, retrieve the evidence behind them, curate deliberately, test against those cases, deploy, and feed the result back into the next model.
Adopt longtail thinking, because once baseline capability becomes good, rare, and compositional states dominate remaining failures.
Applyfailure mining even more aggressively, because robotics policies generate precisely the off-distribution states that expert demonstrations omit.
Carry over curation discipline, because heterogeneous robot data can produce negative transfer as well as positive transfer.
Usesimulation infrastructure, but do not inherit AV assumptions about what can be simulated faithfully.
Extend fleet learning beyond the fleet, to include heterogeneous robots, human demonstrations, simulation, and deployed experience.
What transfers from the autonomous vehicle playbook to general robotics, and what changes in the process.
What transfers from the autonomous vehicle playbook to general robotics, and what changes in the process.
AV practice
Transfers?
What changes in robotics
Long-tail thinking
Almost completely
The unit of rarity shifts from what the sensor sees to state, action, contact, and task history
Failure mining
Transfers, and matters more
Mine recoveries and interventions, not just failures, and preserve the full episode structure
Curation discipline
Transfers, but gets harder
Adds normalization across embodiments, sensors, and action spaces on top of quality filtering
Simulation infrastructure
Transfers
Fidelity must be validated per task. Contact-rich manipulation is far less reliable than locomotion
Fleet learning
Transfers, but not as a fleet
Learning signals come from teleoperation, human video, simulation, and other embodiments
Robotics should preserve that discipline without assuming that the underlying data system will look like an AV stack. The shape of the long tail changes. Failure episodes contain richer action and recovery information. Curation has to account for embodiment and task compatibility. Simulation fidelity varies dramatically by interaction regime. And useful learning signals may come from people, simulations, other robots, and deployed systems rather than a homogeneous fleet. That is the part of the AV playbook worth carrying forward.
The next posts in this series will go one layer deeper into that loop: what data to collect, how to decide that it is valuable, and how to build collection around the failures the deployed policy actually produces.
FAQ
Robotics inherits the data discipline, not the system design. Long-tail analysis, failure mining, curation, simulation infrastructure, and deployment feedback all transfer, but each needs a different implementation because robotics spans many embodiments, tasks, and action spaces.
The unit of rarity changes. In driving perception, a rare event is usually something the camera sees, such as an unusual vehicle or pedestrian configuration. In general-purpose robotics, the tail lives in the combination of state, action, contact, and task history, so the rare event is often how the robot arrived somewhere rather than what it observed.
Selectively. The surrounding infrastructure transfers well, including scenario definitions, deterministic replay, regression suites, and real-to-sim reconstruction. The simulator's authority does not transfer uniformly. Simulation is already reliable for locomotion, navigation, and rigid-body manipulation, but contact-rich and dexterous manipulation should treat it as a source of hypotheses and pretraining data rather than ground truth.
Because heterogeneous robot data can produce negative transfer. DataMIL found that naively adding prior robot data can hurt a specialized downstream policy, while performance-aware selection produced more consistent gains. Curation in robotics means finding the data mixture that improves one capability without degrading others.
