How Much Data Does Your Robot Policy Actually Need?
Author
Technical Writer
Mohamad Louai Shehab is a postdoctoral researcher in the Robotics Department at the University of Michigan.
See all articles by Mohamad Louai Shehab
In this article
Try Voxel51 for yourself
Key takeaways
- More hours of robot training data don’t reliably raise policy success rate. Diversity across scenes, objects, and tasks matters more, and Lin et al. found little gain past about 50 demos per environment.
- Robotics scaling laws exist, but they predict validation loss, which tracked success in EgoScale’s setting but showed no correlation with real-world success across 16 ABC checkpoints (r = −0.04).
- Human-in-the-loop intervention data is one of the strongest levers for robot policy success. In one manufacturing case, 240 corrected rollouts lifted a 1-hour policy from 28% to 88%, beating a 21-hour baseline.
- Curating robot training data by estimated influence can beat training on all of it. CUPID matched state-of-the-art diffusion policies in simulation with under a third of the data.
- Score robot training data with embeddings, trajectory diversity metrics, and influence estimates instead of counting hours.
Is it 2026, or the 1950s?
If you stripped away the modern camera resolution, a 2026 robotics demo from a top lab might look shockingly similar to archival footage from the 1950s. Yes, 1950s, no typo there! Early pioneers succeeded at highly scripted physical tasks, but scaling beyond rigid, hand-crafted routines proved surprisingly elusive. As Deepak Pathak (co-founder and CEO of Skild AI) puts it, robotics has been stuck for 70 years [1].
Meanwhile, the rest of AI has changed beyond recognition. Today’s frontier models have learned to see, talk, write good code, reach gold-medal level at the International Mathematical Olympiad [26], and even solve open problems in mathematics [27].
This disparity usually gets attributed to a single bottleneck: data. Language models ate the internet, trillions of tokens of it. Robotics, on the other hand, got a handful of labs, some teleoperation (teleop) rigs, and a lot of grad students in virtual reality (VR) headsets. So if robotics is going to have its own “GPT moment,” the obvious first fix seems to be collecting more data. But how much more?
To make things concrete, say that 10 hours of demos gets your policy to a 50% success rate. Will 20 hours get you to 100%? In language modeling, scaling laws let you predict that curve before you collect a single extra token. Robotics doesn’t have that curve yet. It’s starting to get something like it (more on that next), but not one you can plan around. So teams fall back on the only lever they can count: more hours.
This piece is about what to do instead of guessing. The short version: stop counting hours, and start measuring what’s in them.
Robot data scaling laws exist, just not the ones you need
As promised, robotics does have “scaling laws” now. The unfortunate news is that they just don’t directly answer the question above.
Perhaps the cleanest comes from NVIDIA’s EgoScale [2, 18]. They trained on 20,854 hours of egocentric human video, and validation loss fell along a clean log-linear curve (see Figure 1). There are two catches. First, the curve tracks validation loss, not success rate. EgoScale reports that the two correlate in its setting, but elsewhere in imitation learning, validation loss has proven an unreliable stand-in for success. Validation loss kept climbing while the policy improved [9], or while success stayed flat [13]. In the ABC behavior cloning study, it showed no correlation with real-world success across 16 checkpoints, even though training loss and validation action error did [4] (see Figure 2). Second, the data is human video, a more niche source than the teleoperated robot data most teams actually collect. Whether the same curve holds for teleop data is still an open question, and whether a similar curve exists for success rate directly is also unclear.
Lin et al., by contrast, measured success directly, across 40,000 demonstrations [3]. Performance followed a power law in the number of distinct environments and objects, that is, “diversity,” not the number of demos. Past about 50 demos per environment, more of the same barely helped. The catch is that datasets generally don’t even define “scene” or “environment” the same way [11], so environment diversity isn’t that easy to calculate. For example, is a new tablecloth considered a new scene? (More on measuring this below.)
So the laws we have either predict a proxy that doesn’t reliably track success, hold for one kind of data, or depend on a variable that’s hard to measure. They do agree on one thing: up to a point, how varied your data is matters more than how much of it you have.
You can’t brute-force your way there
Even if hours were the right unit, you couldn’t buy enough of them.
To put the gap in human terms, it would take an average person about 100,000 years to read all the text used to train today’s large language models. The largest robot datasets add up to roughly one year [5, 6]. That’s five orders of magnitude, and the field won’t close it just by working harder.
The reason is cost. Text is a byproduct of people living online. Every post, paper, and forum thread is training data that nobody had to collect. Robot data has to be made deliberately, one episode at a time, and every minute of it needs a robot, a trained operator, and a reset between attempts.
Much of what exists is also narrow. The large majority of tasks in Open X-Embodiment, one of the field’s largest robotics datasets, are variations of “pick up A, put it in B” [6, 7], a problem that classical vision, inverse kinematics, and motion planning already handle well.
Piling on similar data doesn’t necessarily help, and it can even hurt. In a controlled simulation study, a 10,000-episode dataset outperformed a 50,000-episode one drawn from the same distribution on one task [9].
The same imbalance shows up inside individual demonstrations. Most frames capture free-space motion as the arm cruises toward an object. The moments where policies actually fail, the grasp, the handover, the insertion, make up only a thin slice of each episode.
Simulation and synthetic data are a big part of the answer, and one of the most active threads in physical AI right now. They already pull real weight: in ABC, simulated evaluations tracked real-world success closely enough to compare policies before touching hardware [4]. But simulation isn’t yet a substitute for contact-rich, real-world failure modes. Friction, deformation, and slip are hard to model faithfully, and they’re often exactly where policies break. We covered why in our earlier piece on the sim-to-real gap [8].
So waiting for data alone to deliver robotics its GPT moment isn’t a plan. The useful question is what has actually worked with the data we can afford.
What actually moves robot policy success rate
If simply more hours is the wrong answer, what should you do instead? Six levers come up again and again in recent work, often worth tens of percentage points. One especially useful case study comes up throughout: Dream Machines’ write-up on fine-tuning π0.5 for a manufacturing task [13]. It covers a single task, so its numbers work best as concrete examples alongside the broader research.
Scene, object, and task diversity. This is perhaps the best-supported lever. Lin et al. collected data across 32 environments, each with one object and about 50 demos, and their policies reached roughly 90% success in new environments with unseen objects [3]. For two new tasks, four people collecting for a single afternoon gathered enough data to hit that mark. The pattern holds one level up as well: Shi et al. found that task diversity in pretraining mattered more than the number of demos per task [12]. In the Dream Machines case, four hours spread across five scenes beat four hours collected in the evaluation scene itself, 63% to 33% [13].
How you collect. The people and hardware behind the data shape what the policy learns. Different operators solve the same task in different styles and at different speeds, and that variation can actively confuse policy learning. Across studies, more consistent demonstrations lead to better policies [4, 10, 12]. That doesn’t mean throwing data away. In ABC, training only on the best operator’s demos did worse than using everyone’s. What worked was tagging each demo with who collected it, then asking the policy to imitate the best operator at test time: 8 of 10 successful T-shirt folds versus 4 for the untagged baseline [4]. The interface matters too. In the Dream Machines case, VR teleoperation data reached 61% versus 30% for GELLO leader-arm data [13, 14]. One caveat: each interface had a different operator, so the gap reflects both the tool and the person using it.
Careful, consistent demos. This aims at fixing the intra-demo imbalance issue. ABC’s box-folding policy went from zero success to 24% after 10 hours collected under a stricter operating procedure [4]. In the Dream Machines case, one extra hour of slow, consistent demos (deliberately slowed at the critical steps) on top of 21 hours lifted success from 76% to 90% [13].
Curation. Not all data helps, and some of it hurts, as we hinted earlier. CUPID scores each demonstration by its estimated influence on success. Training on under a third of the data, chosen this way, matched state-of-the-art diffusion policies in simulation [15]. DataMIL applies a similar idea to large public datasets. Picking just 0.5–1% of Open X-Embodiment by estimated benefit roughly doubled real-robot success compared with picking the same amount at random, 61% to 31% [20, Table 1].
Intervention data. Human correction (or human-in-the-loop) is among the most reliable levers for policy improvement. Rather than relying on pure imitation, which is prone to compounding errors, DAgger (Dataset Aggregation) introduced data collection on the actual states a policy visits [16]. Human-gated DAgger (HG-DAgger) refined this by having humans intervene when a robot falters [17]. The impact of these interventions is drastic. In ABC, 10 extra hours of demos got box folding to only 24%. Rather than collecting more, the team ran short DAgger rounds focused on recovery behaviors, and that is what made the task work [4]. In the Dream Machines experiment, 240 human-corrected rollouts boosted a 1-hour policy from 28% to 88% success, comfortably beating a 21-hour baseline [13]. Physical Intelligence’s pipeline scales this same loop. It combines demonstrations, human corrections, and the robot’s own autonomous experience, and more than doubles throughput on its most difficult tasks [19]. Where you intervene matters too. Mulligan found that failures concentrated in a small set of starting configurations, so uniform collection spent much of the operator’s time on states the policy already handled. Starting each collection round at observed failures and untried states beat uniform collection at the same budget [25].
Bootstrap with engineering. You don’t have to start from zero. Whenever you can, deploy a system that’s good enough to be useful, built with classical engineering where needed, and then let its rollouts become training data. Ambi Robotics reports logging 22 years of robot data in four years this way [5].
| Lever | Example result | Source |
|---|---|---|
| Scene, object, and task diversity | 4 hours across 5 scenes beat 4 hours in the evaluation scene, 63% vs. 33% | Dream Machines [13] |
| How you collect | Tagging demos by operator: 8 of 10 T-shirt folds vs. 4 untagged | ABC [4] |
| Careful, consistent demos | 1 extra hour of slow demos on top of 21 hours: 76% to 90% | Dream Machines [13] |
| Curation | Under a third of the data matched state-of-the-art diffusion policies in simulation | CUPID [15] |
| Intervention data | 240 corrected rollouts: 28% to 88%, beating a 21-hour baseline | Dream Machines [13] |
| Bootstrap with engineering | 22 years of robot data logged in four years | Ambi Robotics [5] |
None of these levers is “collect more of the same.” Each one changes what’s in the data, not how much of it there is.
How to measure robot training data quality
The earlier sections keep circling back to things an hour count can’t capture: how diverse the data is, whether more of the same is still helping, which demos actually help, and where the policy fails. Acting on any of them means scoring your data, not just counting it. There are three practical ways to score every episode, and each answers a different question.
Embeddings: what you have and what you’re missing. Map every episode into a shared space with a pretrained encoder. Redundancy shows up as tight clusters. Gaps in coverage show up as empty regions: scenes, lighting, or object poses you never collected. Embed states and actions, not just images. SCIZOR removes redundant state-action pairs this way and gets better policies from less data [22], and DemInf uses similar embeddings to separate high- and low-quality demos [23].
Trajectory metrics: how varied the motions are. Embeddings capture what the robot saw. Trajectory metrics capture what it did. Sirigiri et al. compute a model-free diversity score directly on demonstration trajectories, and curating for it consistently beats random selection [21]. Geometric Entropy adds a ceiling: past a point, more varied motions make the right action ambiguous. In their experiments, extra diversity in the fine-tuning set mostly hurt a pretrained vision-language-action (VLA) model [24].
Influence estimates: which episodes actually help. Looking alike isn’t the same as helping alike. In DataMIL, the most helpful and most harmful samples often looked similar, because similar states can carry very different actions [20]. Influence methods skip appearance and estimate each episode’s effect on the policy directly. CUPID does it from evaluation rollouts [15], and DataMIL from a proxy loss that needs no rollouts at all [20].
If you want to see what this looks like before setting anything up, the Voxel51 free plan includes the LeRobot Community dataset on an embeddings plot, no credit card required.
A better answer than “more hours”
So, back to the opening question: if 10 hours gets your policy to 50%, will 20 get you to 100%? Nobody can tell you yet, and the research suggests the hour count was never the right thing to plan around.
What drives progress is what’s in the data. Spread it across scenes and objects instead of piling it into one setup. Keep demonstrations consistent, and slow down where the task is hard. Curate out what doesn’t help. When the policy fails, put a human in the loop at those moments instead of starting over with fresh demos. Each of these is cheaper than another week of teleoperation, and you can check each one in your data before you spend the robot time. When you’re ready to look at your own episodes, Voxel51 is free to try, with nothing to install.
Robotics won’t get past pick-and-place by collecting more of the same data. It will get there by collecting the data its policies are missing.
Frequently asked questions
No one can give a reliable hour count yet, because robotics has no scaling law that predicts success rate from data volume. Lin et al. found that diversity across environments and objects predicts success better than demo count, with little gain past about 50 demos per environment.
Validation loss doesn’t reliably predict robot policy success. In the ABC study it showed no correlation with real-world success across 16 checkpoints (r = −0.04), while training loss (r = −0.84) and validation action error (r = −0.89) tracked it.
Human-in-the-loop intervention data is among the most reliable levers. In the Dream Machines case, 240 human-corrected rollouts raised a 1-hour policy from 28% to 88% success, beating a 21-hour baseline.
Robot training data quality can be scored three ways: embeddings for coverage and redundancy, trajectory metrics for motion diversity, and influence estimates for which episodes help the policy.
Sources
- Deepak Pathak (Skild AI). Robotics Has Been Stuck for 70 Years. Talk, AI Engineer, YouTube, 2026. https://www.youtube.com/watch?v=jFHteJjRl8A
- Ruijie Zheng, Dantong Niu, Yuqi Xie, Jing Wang, Mengda Xu, Yunfan Jiang, Fernando Castañeda, Fengyuan Hu, You Liang Tan, Letian Fu, Trevor Darrell, Furong Huang, Yuke Zhu, Danfei Xu, Linxi Fan. EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data. arXiv, 2026. https://arxiv.org/abs/2602.16710
- Fanqi Lin, Yingdong Hu, Pingyue Sheng, Chuan Wen, Jiacheng You, Yang Gao. Data Scaling Laws in Imitation Learning for Robotic Manipulation. ICLR 2025.
- Arthur Allshire, Himanshu Gaurav Singh, Ritvik Singh, Adam Rashid, Hongsuk Choi, David McAllister, Justin Yu, Yiyuan Chen, Huang Huang, Pieter Abbeel, Xi Chen, Rocky Duan, Phillip Isola, Jitendra Malik, Fred Shentu, Guanya Shi, Philipp Wu, Angjoo Kanazawa. Scalable Behavior Cloning with Open Data, Training, and Evaluation (ABC). arXiv, 2026. https://arxiv.org/abs/2606.27375
- Ken Goldberg. Good Old-Fashioned Engineering Can Close the 100,000-Year "Data Gap" in Robotics. Science Robotics, 2025.
- Ken Goldberg. A Tale of Two Cultures: Can Agentic Coding Close the Gap? ICRA 2026 plenary.
- Open X-Embodiment Collaboration. Open X-Embodiment: Robotic Learning Datasets and RT-X Models. ICRA 2024.
- Mohamad Louai Shehab. On the Sim-to-Real Gap. Voxel51 blog, October 22, 2026. https://voxel51.com/blog/sim-to-real-gap-diagnosis
- Seohong Park. Behavioral Cloning Mystery. Blog post, 2026. https://seohong.me/blog/behavioral-cloning-mystery/
- Suneel Belkhale, Yuchen Cui, Dorsa Sadigh. Data Quality in Imitation Learning. NeurIPS 2023.
- Alexander Khazatsky et al. DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset. RSS 2024.
- Modi Shi, Li Chen, Jin Chen, Yuxiang Lu, Chiming Liu, Guanghui Ren, Ping Luo, Di Huang, Maoqing Yao, Hongyang Li. Is Diversity All You Need for Scalable Robotic Manipulation? arXiv, 2025.
- Dream Machines. Empirical Results Fine-Tuning π0.5 on a Real Manufacturing Task. Blog post, 2026. https://dream-machines.eu/blog/pi05-fine-tuning
- Philipp Wu, Yide Shentu, Zhongke Yi, Xingyu Lin, Pieter Abbeel. GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators. arXiv, 2023.
- Christopher Agia, Rohan Sinha, Jingyun Yang, Rika Antonova, Marco Pavone, Haruki Nishimura, Masha Itkina, Jeannette Bohg. CUPID: Curating Data Your Robot Loves with Influence Functions. CoRL 2025.
- Stéphane Ross, Geoffrey J. Gordon, J. Andrew Bagnell. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (DAgger). AISTATS 2011.
- Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell, Mykel J. Kochenderfer. HG-DAgger: Interactive Imitation Learning with Human Experts. ICRA 2019.
- Jim Fan (NVIDIA). Robotics: The End Game. Keynote, MACHINA 2026, Paris, July 7, 2026.
- Physical Intelligence, Ali Amin et al. π*0.6: a VLA That Learns From Experience. arXiv, 2025.
- Shivin Dass, Alaa Khaddaj, Logan Engstrom, Aleksander Mądry, Andrew Ilyas, Roberto Martín-Martín. DataMIL: Selecting Data for Robot Imitation Learning with Datamodels. arXiv, 2025.
- Sreevardhan Sirigiri, Nathan Samuel de Lara, Christopher Agia, Florian Shkurti, Fabio Ramos. Diversity You Can Actually Measure: A Fast, Model-Free Diversity Metric for Robotics Datasets. arXiv, 2026.
- Yu Zhang, Yuqi Xie, Huihan Liu, Rutav Shah, Michael Wan, Linxi Fan, Yuke Zhu. SCIZOR: A Self-Supervised Approach to Data Curation for Large-Scale Imitation Learning. ICRA 2026.
- Joey Hejna, Suvir Mirchandani, Ashwin Balakrishna, Annie Xie, Ayzaan Wahid, Jonathan Tompson, Pannag R. Sanketi, Dhruv Shah, Coline Devin, Dorsa Sadigh. Robot Data Curation with Mutual Information Estimators (DemInf). RSS 2025.
- Qian Luo, Ruizhe Liu, Pei Zhou, Xunzhe Zhou, Yanchao Yang. Geometric Entropy: When Trajectory Diversity Helps and Hurts in Imitation Learning. arXiv, 2026.
- Lars Ankile, Perry Dong, Rohan Bhowmik, Aneesh Muppidi, David D. Yuan, Shuran Song, Chelsea Finn. Mulligan: Performance-Guided Data Collection for Efficient On-Robot Learning. arXiv, 2026.
- Google DeepMind. Gemini Deep Think achieves gold-medal standard at the International Mathematical Olympiad, 2025. https://deepmind.google/blog/advanced-version-of-gemini-with-deep-think-officially-achieves-gold-medal-standard-at-the-international-mathematical-olympiad/
- Davide Castelvecchi. AI cracks 80-year-old mathematics challenge — researchers are astonished. Nature, 2026.
Technical Writer
Mohamad Louai Shehab is a postdoctoral researcher in the Robotics Department at the University of Michigan.
See all articles by Mohamad Louai Shehab
Try Voxel51 for yourself
Loading related posts...
© 2026 Voxel51 All Rights Reserved
