The robot data stack is consolidating layer by layer. Quality scoring is commoditizing fastest, this year's audits show those scores can invert on the errors that matter, and the layer nobody owns is the one that turns a score into a decision.
Over the past eighteen months the same short list of quality metrics for robot demonstrations has been reimplemented over and over: spectral smoothness, jerk, idle-timestep detection, actuator saturation, timestamp and sensor health. It shows up in papers from unrelated labs, in a wave of CLI scorers and converters appearing on GitHub, and in the internal QA script that every robotics team writes in its first year.
When that many people build the same thing at the same time without coordinating, two things are true at once. The need is real, and the thing is about to become a commodity.
Controlled audits published this year found that action-only quality scores miss the errors that actually degrade trained policies, and that some of them invert: they rank defective demonstrations above clean ones. Everyone is scoring. Almost nobody has built the part that makes a score worth acting on.
Key takeaways
The log-format war is over: MCAP has been the default ros2 bag record output since ROS 2 Iron, and its schemas live inside the file, so recordings stay decodable without a ROS install.
The training-format war is not over. LeRobot, RLDS, HDF5, and Zarr each carry their own model ecosystem, which is why conversion scripts are the most duplicated code in robot learning.
Cheap smoothness scoring works and is nearly free. A single DFT per demonstration ranked a 300-episode RoboMimic task in 10 seconds, against 300 to 360 minutes for the policy-in-the-loop methods it was compared to (arXiv:2605.01544).
On a structural defect (an early gripper release), smoothness scored defective demonstrations better than clean ones, at an AUROC of 0.244 against a 0.5 chance line, and policies curated with it trended below the uncurated baseline (arXiv:2606.05588).
Detection accuracy does not predict curation value. On a contact-rich LIBERO benchmark, the best detector (AUROC 0.804) produced a 13.3% policy while a mediocre one (AUROC 0.638) produced 90.0% against a 93.3% oracle. Rank correlation between the two axes: -0.14 (arXiv:2606.10229).
Quality scores are triage, not gates. The scarce layer is the loop: score, see the flagged moment in synced playback, decide reversibly.
Robot data is speedrunning the image machine learning data stack
Image ML went through this. Roughly 2015 to 2020, the layers settled in order. First the format fight ended, with COCO conventions winning detection by being good enough and everywhere. Then labeling tooling matured. Then quality and curation tooling arrived: label-error detection, near-duplicate removal, coverage analysis. Then platforms fused the layers, because the value was in the workflow rather than in any single layer.
Robot learning data is running the same sequence, compressed into about two years, on harder data.
Harder in specific ways. Each recording is an episode, not an independent sample: many channels ticking at different rates on a shared clock. Quality is a property of a trajectory over time, so there is no frame you can look at to judge it. And the failures are silent. The Boreas Road Trip release documents an IMU whose clock drifted enough that timestamps had to be recomputed from sensor-reported message counts, with dropout periods as long as 0.4 seconds, long enough that multiple camera and LiDAR frames land with no inertial data behind them at all. KITTI documented a worst-case 5 ms offset between a GPS/IMU reading and its paired camera and Velodyne package, because the inertial unit could not be hardware-triggered off the spinning lidar the way the cameras could.
That is what recording six sensors on one vehicle looks like, and it is invisible in every summary statistic anyone computes over the dataset.
So the stack is assembling itself, layer by layer, around data with those properties. Worth mapping where it stands.
The robot data stack, mapped
The robot data stack layer by layer, from the settled log-format war to the unclaimed verification loop.
The robot data stack layer by layer, from the settled log-format war to the unclaimed verification loop.
Layer
What it does
Occupants
State
Log format
Records on the robot
MCAP, legacy rosbag, ULog
Settled, MCAP won
Training format
Feeds the dataloader
LeRobot, RLDS, HDF5, Zarr
Active format war
Conversion
Log format to training format
Converters on GitHub, internal scripts
Fragmented, heavily duplicated
Analysis
Gets a channel into a dataframe
Ad hoc scripts, dataframe wrappers
Fragmented
Quality scoring
Ranks and flags episodes
Research code, new CLI scorers monthly
Commoditizing in real time
Cataloging and search
Queries across a corpus
Commercial platforms, embedded indexes
Contested, commercially funded
Visualization
Shows you one episode
Foxglove, Rerun, FiftyOne
Three strong players
Verification loop
Score, see, decide
Unclaimed
Nearly empty
A few notes on the rows that matter.
The log layer settled because MCAP put each channel's schema inside the file. A recording stays readable years later by a tool that has never heard of your message packages, which is why it became the ROS 2 default bag format starting with Iron.
The training layer has not settled, and it will not settle soon, because each format is load-bearing for a different model lineage: LeRobot for the Hugging Face stack, RLDS for the Open X-Embodiment and OpenVLA line, HDF5 for ALOHA and robomimic, Zarr for Diffusion Policy and UMI. Every pair of those formats needs a converter, and everyone writes their own.
The three names in the visualization row are not competing for the same job. Foxglove is debugging-first: one log, live connections, deep protocol fluency. Rerun is logging-first: instrument your code, stream what it sees, watch it as it runs. Both are excellent at that. The dataset-first job is a different shape: ten thousand episodes with fields, queries, embeddings, and tags, where the question is which of those logs you should be looking at.
And then the last row, which is what this post is about.
Robot data quality scoring is commoditizing, and the research says the need is real
Automated scoring exists because manual review lost, and it lost on arithmetic.
DROID is 76,000 demonstration trajectories, 350 hours of interaction, 564 scenes, 86 tasks, collected by 50 people across 13 institutions over 12 months. Open X-Embodiment pools roughly 60 datasets across 22 embodiments. At that scale most of what gets collected does not survive filtering: SayCan collected 276,000 autonomous rollouts and kept 12,000 after success filtering and human review, and the audit work below assumes 20 to 40 percent contamination is typical of a real pipeline.
The variance is not random. Mandlekar et al. showed that operator skill accounts for a large share of the spread in downstream policy performance (arXiv:2108.03298), and Belkhale et al. decomposed data quality into action divergence and transition diversity (arXiv:2306.02437). Behavior cloning compounds error, so a little bad data poisons a lot of policy.
The metrics converged because they are cheap and they demonstrably move the number.
Smoothness scoring is imported wholesale from motor-control and rehabilitation research, where spectral arc length and log dimensionless jerk have been validated as movement-quality measures for over a decade (Balasubramanian et al., IEEE TBME 2012; review in J NeuroEng Rehabil 2015; reference implementation). RINSE ported spectral arc length to demonstration curation and, on RoboMimic Transport, trained a better policy on the smoothest 50 of 300 demonstrations than on all 300: 55% success against 39% (arXiv:2604.23000). On a real xArm push task, its contact-aware variant hit 88% with 100 of 200 demonstrations against 68% on the full set.
The cheapness is the point. A separate group ranked demonstrations by total spectral power, one DFT per trajectory, no labels, no policy, no environment interaction. Ranking a 300-demonstration RoboMimic task took 10 seconds, against 300 and 360 minutes for the two learned baselines they compared against. Then they fine-tuned a pi-0.5 policy with LoRA on demonstrations collected by five older adults at a retirement facility, keeping 60% of each participant's episodes, and mean success went from 15% to 38% (arXiv:2605.01544).
There is an expensive tier for teams that already have a policy: mutual information estimates (DemInf beat full-data training by more than 10 points on RoboMimic Can, arXiv:2502.08623), influence functions (CUPID improved a real Franka figure-8 task by 38 points, averaged over its filtering and selection settings, arXiv:2506.19121), and datamodels (DataMIL). The result that keeps replicating across all of it is that training on less, chosen well, beats training on everything.
Now the commoditization point. The cheap tier is a DFT and some summary statistics. Every scorer on GitHub and every internal QA script computes some subset of the same list, and within a year "we compute spectral arc length" will differentiate nobody. Nobody competes on computing precision and recall either. The metric layer always commoditizes. The workflow layer is where the value pools.
The scores cannot be trusted on their own
Two audits published this year did the obvious experiment that nobody had done: inject defects with known type and location, hide the labels from every metric, and then score each metric twice. Once on detection, and once on whether a policy trained on what it kept actually succeeds more often.
The results split cleanly by defect type.
Subtle perturbations behave sensibly. Correlated action noise, tremor, and truncation are caught by multivariate outlier scoring at an AUROC of 0.968, and removing them recovers the whole downstream gap.
Structural errors do not. When the defect is a wrong action at a decisive moment (the gripper commanded open partway through the carry, so the object drops), every action-only metric fails, and two of them fail in the wrong direction. Smoothness scored the defective demonstrations as better than the clean ones, an AUROC of 0.244 where 0.5 is chance, and a policy curated by smoothness reached 27.3% success in point estimate against 36.0% for no curation at all. Isolation forest, the winner on the subtle regime, sat at 0.543, indistinguishable from a coin flip. Only the metrics that read the state trajectory detected it, and the best of those recovered about a third of the 27-point gap to the oracle (arXiv:2606.05588).
The companion study ran the same defect on a contact-rich LIBERO pick-and-place and found that detection and downstream value are uncorrelated. The metric with the highest detection AUROC (0.804) produced the worst curated policy at 13.3% success, while a metric with a much lower AUROC (0.638) produced 90.0% against a 93.3% oracle ceiling. Spearman correlation between detection rank and policy rank across seven metrics: -0.14. The same paper found that five of the seven metrics had been quietly reading episode length as a proxy for the defect label, scoring a perfect 1.000 AUROC before length was controlled and dropping to chance after (arXiv:2606.10229).
Keep it honest. Both audits are recent preprints from one author, run on injected defects in simulation with three seeds, and the papers say so plainly in their own limitations sections. Read the downstream magnitudes as preliminary. The qualitative finding is the durable part, and it has a mechanism behind it rather than just a p-value.
The mechanism is simple. A wrong-but-smooth action looks perfect to every metric that only reads actions, because the statistics of the motion are fine. What the wrong action changes is where the robot ends up: in states no good demonstration ever visits. You can see that in the state trajectory, or you can see it by looking at the episode. You cannot see it in the action spectrum.
Which gives the design rule: automated scores are triage, never a gate. A score's job is to decide what a human looks at first, not to decide what gets deleted. Any pipeline that wires a smoothness threshold directly to a delete button is, on the published evidence, sometimes deleting its best data and keeping its worst.
The same argument applies to the other half of quality that almost nobody scores: sensor health. The drone world institutionalized this years ago. PX4 Flight Review automatically checks vibration spectra, GPS jamming indicators, estimator watchdog flags, and logging regularity on every uploaded flight, down to whether the interval between logged IMU samples departs from the expected 4 ms. General robotics mostly still finds its clock drift by accident, three weeks into a bad training run.
The missing layer is the verification loop
Look at where the tooling ends today. Scores land in a CLI table or a JSON sidecar. Verification, if it happens, means scrubbing a viewer that knows nothing about the scores. Curation means a script that writes out a new copy of the dataset. The practitioner's workflow crosses every layer of the stack, and the tools terminate at the layer boundaries.
Closing the loop takes five things.
Scores live next to the data. Episode-level metrics as queryable fields on the dataset, not a sidecar CSV. Sort worst-first, threshold, and combine with any other field.
Every flag is two clicks from evidence. A flagged interval deep-links into synced multimodal playback at the offending timestamp. The moment that matters is: the score said bad, and my eyes confirm bad.
At least one state-aware signal. Action-only suites are structurally blind to the defects that hurt most. Something manifold-based belongs in the default set.
Decisions are reversible metadata. Tag exclusion candidates, save the view, export the survivors. Thresholds get revised, so never delete.
Provenance and reproducibility. Metric config versioned with the run, and scores from different configs never silently compared.
That loop is the natural hub of the stack, and the audits make it an empirical claim rather than a preference: no score has earned the right to decide alone. Whichever surface owns the looking is where scores from every tool and every paper become actionable.
The pieces exist separately: viewers play anything, scorers score anything, dataset platforms query anything. The fusion is missing, and it is not a small feature. It needs synced episodic playback, a field and query system, embeddings, and tagging in one place. Playing heterogeneous sensors back on one clock is a rendering problem before it is a data problem, which is why this is years of infrastructure for anyone starting from zero.
What to do about it
If you are running a robot data pipeline, the staged version is short.
Automate the cheap universal layer this quarter: smoothness, idle, saturation, timestamp and sensor health on every episode. No labels or policies required.
Wire those scores to inspection before you wire them to anything else. Triage, not gates.
Put at least one state-aware metric in the set.
Treat curation as tagging and views rather than deletion.
Graduate to policy-in-the-loop methods only when a policy exists and the cheap layer is exhausted.
Where we fit: FiftyOne 1.19 added native MCAP ingestion and synced multimodal episodic playback, so cameras, point clouds, GPS, IMU plots, and logs open on one clock next to your images and video (launch details here). That supplies the substrate the loop needs: fields, queries, embeddings, temporal tags, and a synced viewer in one place. We have been building the verification loop described above on top of it as an open-source plugin. It scores episodes, writes the scores back as queryable fields, and deep-links each flagged interval to its timestamp in the viewer.
The metrics were never going to be the product. The stack will keep consolidating, the scores will keep commoditizing, and the teams that win the data race will be the ones who can go from flagged to seen to decided in seconds, ten thousand times.
The bottleneck was never computing the number. It was looking.
Only some kinds. Smoothness catches subtle perturbations like tremor and noise, but on a structural defect such as an early gripper release it scored defective demonstrations better than clean ones (AUROC 0.244, where 0.5 is chance).
MCAP has been the default ros2 bag record output since ROS 2 Iron. Its schemas live inside the file, so recordings stay decodable without a ROS install.
No. On a contact-rich LIBERO benchmark, detection AUROC and downstream policy success were uncorrelated (Spearman -0.14). The best detector produced a 13.3% policy while a mediocre one produced 90.0%.
No. Scores are triage, not gates. Wire them to inspection first, tag exclusion candidates as reversible metadata, and never wire a threshold directly to deletion.
Because each format is load-bearing for a different model lineage: LeRobot for the Hugging Face stack, RLDS for Open X-Embodiment and OpenVLA, HDF5 for ALOHA and robomimic, and Zarr for Diffusion Policy and UMI.
