Every Synthetic Robot Traces Back to a Real One: Synthetic Data Is a Curation Problem
Aug 24, 2026
• 11 min read
In this article
In this article
Talk to an AI expert
Synthetic data promises an attractive trade for physical AI teams: stop waiting for the physical world to produce enough training data and generate the scenarios you need instead. According to the 2026 State of Visual and Physical AI Survey, 63% of practitioners agree that synthetic data will become the primary source of training data.
A warehouse robot does not have to wait for a pallet to fall at exactly the wrong angle. An autonomous vehicle does not need thousands of physical drives to encounter every combination of weather, lighting, traffic, and road geometry. A manipulation system can train against variations in objects, poses, backgrounds, and illumination without resetting a physical workcell after every attempt.
Good neural reconstruction
Poor-quality neural reconstruction
Synthetic data changes the economics of producing another million frames, trajectories, or scene variations. But generating more data does not eliminate the data problem. More than half of physical AI simulations can end up unusable because of problems in the input data, wasting compute before useful sim data ever reaches training.
However, scale creates another problem: every decision about what goes into a synthetic pipeline can be amplified many times over. Once one demonstration, reconstructed scene, or simulation configuration can produce hundreds or thousands of variations, collecting more data is no longer the only question. Teams also need to decide which observations, scenarios, and configurations are worth amplifying in the first place.
Synthetic generation can scale a seed. It cannot decide whether that seed fills an important gap, duplicates something you already have, or carries a problem into the next million samples. That makes synthetic data generation a curation problem.
Key takeaways
- Synthetic data generation amplifies whatever is in your seed data, so synthetic data curation determines whether you scale useful variation or repeated mistakes.
- Quality, coverage, and uniqueness, applied in that order, tell physical AI teams which seeds are valid, where data is missing, and which examples add new information.
- Uniqueness scoring works best inside a targeted slice of the dataset, not as a global ranking across all samples.
- Generated data needs the same audit as source data, because synthetic samples can fail in ways their seeds did not.
- FiftyOne Physical AI Workbench connects seed auditing, reconstruction, and synthetic generation so real and synthetic samples stay inspectable in one place.
Generation amplifies your data decisions
Synthetic data generation can be considered a multiplication operation. That multiplication is powerful precisely because it is asymmetric: a relatively small decision about the source data can determine the composition of a much larger generated dataset. A recorded pick-and-place demonstration can become hundreds of variations across object positions, lighting, clutter, camera pose, or background. A reconstructed driving scene can be modified across weather, illumination, actors, and viewpoints. Simulation tasks can vary objects, initial states, environments, and trajectories.
The exact “seed” differs by pipeline. In real-to-sim workflows, it may be recorded sensor data. In imitation-learning pipelines, it may be a robot demonstration. In procedurally generated simulation, there may not be a single real example at all—but engineers still define the assets, tasks, parameter ranges, physics, sensor configurations, and distributions from which the synthetic dataset is sampled. In every case, what you choose to vary and what you leave fixed determines what diversity the generated dataset can actually provide.
Robot-learning researchers are increasingly showing how consequential these composition decisions are. The MimicLabs study examined dataset composition across dimensions including camera placement, object type, texture, spatial arrangement, scenes, and robot motions. Across its experiments, alignment and variation in camera poses and spatial arrangements had substantial effects on downstream policy performance. Simply increasing variation in every available dimension did not produce the same benefit.
That is a useful warning against treating domain randomization as synonymous with more randomization. Suppose a manipulation dataset contains thousands of demonstrations of the same task from nearly the same camera geometry. Generating thousands of cosmetic variations may still leave the underlying behavioral coverage largely unchanged. Conversely, a much smaller source set that captures different approach angles, object configurations, failure modes, camera placements, or task states may give the generation pipeline access to more meaningful variation.
The same issue appears at the task level. Synthetic-data efforts have historically emphasized scene-level diversity (such as object instances and poses) more than task-level diversity, which can limit generalization to new robotic tasks.
So the goal of synthetic data curation is not simply to find inputs that can produce a lot of outputs. It is to identify the inputs and configurations that expand the parts of the data distribution that matter downstream. That requires looking at seed data through three lenses: quality, coverage, and uniqueness.
1. Quality: Eliminate bad seeds before they scale
For physical AI, a sample can look perfectly reasonable in an RGB viewer and still be unusable downstream. A multi-sensor sequence may contain incorrect camera intrinsics or extrinsics. LiDAR may be spatially misaligned with the cameras. Sensor streams may be offset in time. Coordinate conventions may disagree. Frames or metadata may be missing. Over 50% of simulations end up unusable because of bad input data—slowing teams and wasting millions in compute costs.
FiftyOne Physical AI Workbench automatically audits input data across more than 75 checks, including sensor calibration, temporal synchronization, LiDAR-to-camera projection, depth-to-LiDAR alignment, coordinate-system consistency, and metadata completeness.
This should be the first gate in seed curation. A corrupted sample can appear highly unusual precisely because it is corrupted. Novelty is not useful when the source itself is invalid. Once invalid data is removed, the harder question becomes whether the remaining pool represents the situations you actually need.
2. Coverage: Map what the seed set represents
A dataset with one million samples can still have poor coverage. Volume is easy to measure while distribution is not. A robotics dataset might contain enormous amounts of nominal operation but very little data for a specific approach angle, object arrangement, lighting transition, interaction state, or failure mode.
Embeddings give teams another way to inspect that composition. Instead of browsing sequential frames or grouping samples only by predefined metadata, teams can project high-dimensional representations of the data into an embedding space. Samples with related visual or semantic characteristics tend to appear near one another, making larger structural patterns easier to inspect.
The important step is determining what’s actually driving those patterns. An embedding cluster is not automatically a scenario, and an empty area in a 2D projection does not automatically identify a deployment gap. Embedding structure should be interrogated alongside task metadata, sensor configuration, labels, model results, and the underlying media.
For synthetic seed selection, this turns embedding exploration into a concrete set of questions:
- Which deployment-relevant modes are already densely represented?
- Which important modes have only a handful of examples?
- Are rare failure states concentrated in specific regions?
- Is apparent diversity coming from changes in scene semantics, or only from nuisance variation such as background or texture?
- Does the candidate seed set cover the camera geometries, object configurations, and task states that matter for the target policy?
The embeddings panel in FiftyOne links representation space back to the actual samples, so engineers can select a cluster or region and inspect the corresponding media rather than reasoning from the projection alone.
The objective is not necessarily to ensure uniform coverage of embedding space. It is to identify the regions that matter to the deployment target and ask whether the candidate seed set covers them at the right depth.
Uniqueness: Decide what seed adds new information
Coverage tells you where the dataset is concentrated. Uniqueness helps separate new information from repetition within that distribution.
FiftyOne can compute uniqueness to score samples according to their relative similarity to the rest of the collection. Scores are normalized from 0 to 1, with higher values representing samples that are more unique relative to the collection. The calculation can use FiftyOne's default embeddings, supplied embeddings, or another model.
For synthetic-data curation, the useful part is not simply sorting the entire dataset from most to least unique. Consider 50,000 robot sequences dominated by the same task and approach angle. A global uniqueness ranking might surface unusual scenes, but the highest-scoring sample could be a sensor artifact, an irrelevant environment, or a task you do not intend to train.
A better approach is to use uniqueness inside a meaningful slice of the dataset. If a target deployment gap is “side-angle grasp attempts on reflective objects,” first create that slice using metadata, semantic search, embeddings, or model results. Then use uniqueness to choose examples that minimize repetition within that scenario.
That changes uniqueness from “find me the weirdest samples” to “within the scenario I care about, which seeds add information I do not already have?”
Quality first. Coverage second. Uniqueness third.
This is where the three signals work together: Quality defines what is valid. Coverage defines where more data is needed. Uniqueness helps decide which examples within those regions deserve amplification.
FiftyOne Dataset Views can turn those decisions into reproducible subsets. Views support chained filtering, searching, slicing, and sorting operations over the source dataset; they store the rules used to construct the subset rather than creating a disconnected copy of the data.
A seed view might therefore encode criteria such as:
Or:
Instead of handing a generator an arbitrary batch of files, teams can start the generation job from a subset with an explicit reason for existing.
| Lens | Question it answers | FiftyOne capability |
|---|---|---|
| Quality | Is the seed valid enough to amplify? | Physical AI Workbench, 75+ automated input checks |
| Coverage | Which deployment scenarios are missing or thin? | Embeddings panel, embedding-space exploration |
| Uniqueness | Which samples in a target slice add new information? | compute_uniqueness, scored 0 to 1 |
Generated data belongs under the same audit
Generation is not the end of curation; the generated output needs to earn its place in the dataset as well. Synthetic samples can fail in ways their seeds did not. A generative model might change appearance while corrupting action dynamics. A simulation sweep might produce thousands of near-identical trajectories. An augmentation intended to address one scenario might inadvertently change another important variable. In other words, generated variation is not necessarily useful variation.
RoboCurate illustrates the problem at the action level. The researchers point out that video generation can produce visually convincing robot demonstrations whose inferred actions are physically inconsistent. Rather than relying only on visual quality, RoboCurate replays predicted actions in simulation and measures whether the simulator motion aligns with the generated video before accepting the trajectory.
Other research on generative robotic simulation similarly argues that evaluation should separate quality, diversity, and generalization, finding that generation methods do not perform uniformly across all three dimensions.
That means the output should return to the same basic audit as the input: Is it valid? Did it add coverage? Is it redundant?
Embedding real and synthetic samples together makes this much easier to see. If a seed cluster represents rainy nighttime intersections, and the generated samples just form a tight cloud around it, you’ve added volume without expanding coverage. If they spread into new regions, that may be useful—but it still needs validation. Those regions could reflect meaningful new combinations, or just artifacts of the generation process. The goal isn’t maximum separation in embedding space. It’s targeted expansion into the scenarios you actually care about.
Lineage matters here too. Synthetic samples should retain enough metadata to answer questions such as:
- Which seed, reconstruction, or simulation scenario produced this sample?
- Which model and generation configuration produced it?
- Which variables were changed?
- Which variables were intentionally held constant?
- Which generation run did it belong to?
Without that lineage, teams can determine that a synthetic sample failed. With it, they can determine which generation decision keeps producing failures.
FiftyOne Physical AI Workbench brings this loop together by unifying dataset exploration, reconstruction, and synthetic generation workflows in a single environment. It enables teams to organize source inputs, simulation controls, and generated outputs within the same dataset structure, so that real and synthetic samples can be inspected side by side. Through direct integrations with NVIDIA Omniverse NuRec and NVIDIA Cosmos world foundation models, it turns synthetic generation into a closed-loop process where outputs are continuously evaluated against the original data rather than treated as standalone artifacts.
The loop becomes: Real data → curate seeds → generate → audit synthetic output → train and evaluate → identify new gaps → curate again.
Synthetic data curation starts with better seed selection
Synthetic data removes one of physical AI's hardest constraints: the need to physically encounter every scenario you want a model to learn from. However, it does not remove the need to decide what the model should see.
As generation becomes cheaper, that decision becomes more important, not less. Ten redundant seeds can produce 100,000 redundant variations just as easily as ten useful seeds can expand an underrepresented part of the operating domain.
The synthetic-data pipeline therefore should not begin with “what can we generate?”
It should begin with the dataset: What are we missing? Which real observations or simulation configurations best represent that gap? Are they trustworthy? And after generation, did we actually close it?
FiftyOne provides the core mechanics for this loop through embedding-based dataset exploration, uniqueness analysis, and dataset views that let teams move fluidly between slices of real and generated data. On top of that, the FiftyOne Physical AI Workbench extends this data-centric workflow upstream into simulation-ready data auditing, enrichment, reconstruction, and synthetic-data generation pipelines.
The result is a synthetic data pipeline that is no longer driven by volume, but by intent. Every generated sample traces back to a deliberate choice about what the model still needs to learn, and where each new batch of data measurably improves coverage of the real-world conditions your system will face.
FAQ
Synthetic data curation is the practice of selecting, validating, and auditing the seed data that synthetic generation pipelines amplify, plus the generated outputs they produce. It treats generation as a multiplier: what you choose to vary and what you leave fixed determines what diversity the generated dataset can provide.
Most unusable simulation output traces back to problems in the input data, such as incorrect camera intrinsics, misaligned LiDAR, time-offset sensor streams, or missing metadata. Voxel51 customers report that over 50% of simulations end up unusable for these reasons.
Apply three lenses in order: quality (is the sample valid?), coverage (does the seed set span the deployment scenarios that matter?), and uniqueness (does this example add information you don't already have?).
Yes. Generated samples can be visually convincing while carrying physically inconsistent actions or near-duplicate trajectories, so outputs should return to the same audit as inputs: is it valid, did it add coverage, and is it redundant?
Talk to an AI expert
Loading related posts...
© 2026 Voxel51 All Rights Reserved
