From Fleet Log to Simulation-Ready Scenario: The Data Layer Behind an AV Data Flywheel
Author
Head of Physical AI Partnerships
Building the future of Physical AI at Voxel51 🚀
See all articles by Dan Gural
In this article
Key takeawaysHow the AV data flywheel loop worksFind: Semantic search over video, where the data already livesCheck: Don't spend GPU hours on a broken driveGenerate: Every synthetic clip knows where it came fromReconstruct: NuRec neural reconstruction you can trustEvaluate: One number can lieAt Porsche and the Volkswagen GroupWhat carries overQuestions? We have answers.
In this article
Key takeawaysHow the AV data flywheel loop worksFind: Semantic search over video, where the data already livesCheck: Don't spend GPU hours on a broken driveGenerate: Every synthetic clip knows where it came fromReconstruct: NuRec neural reconstruction you can trustEvaluate: One number can lieAt Porsche and the Volkswagen GroupWhat carries overQuestions? We have answers.
Try Voxel51 for yourself
The scenario that breaks a driving stack is rarely sitting in a folder labeled "edge cases." It's eight seconds of an ego vehicle overtaking a truck towing a trailer, buried in a six-camera log that runs for hours. Finding it is one problem. Turning it into something you can reconstruct, vary, and retest is four more.
At GTC Berlin, Porsche, the Volkswagen Group, and CARIAD announced a shared autonomous vehicle (AV) data flywheel with NVIDIA, Microsoft, and Voxel51, with Porsche leading the initial implementation. This post covers the technical side of Voxel51's part: how the Voxel51 platform connects NVIDIA Cosmos and Omniverse NuRec into one loop, what each stage does, and how every scene stays traceable from the drive it came from to the simulator it lands in.
Key takeaways
- The Voxel51 platform connects NVIDIA Cosmos Dataset Search, Cosmos Data Factory, and Omniverse NuRec into one loop that keeps a single record per driving scenario, from search to evaluation.
- Auditing raw MCAP recordings before neural reconstruction catches calibration, timing, and actor defects that would otherwise produce a NuRec run that scores fine and looks wrong.
- Stored gate evidence for Cosmos-generated clips showed that 12 sampling steps matched 35 on accept rate, and it let the team recalibrate the judge with no new GPU time.
- PSNR alone can pass a poor neural reconstruction: one nuScenes night scene scored 28.59 dB PSNR with 0.56 SSIM and 0.61 LPIPS.
- Every NuRec reconstruction lands in a human review queue as accepted, sent back for retraining, or rejected, with its audit and evidence attached.
How the AV data flywheel loop works
The loop has five stages: find, check, generate, reconstruct, and evaluate. Every stage is a handoff, and every handoff is where context gets lost. A clip found in search loses its calibration. A synthetic clip loses its link to the real one it came from. A reconstruction score loses the drive it was scored on. Our job is to keep one record per scenario through the whole loop:
| Stage | NVIDIA component | What Voxel51 does |
|---|---|---|
| Find | Cosmos Dataset Search | Runs semantic video search from the dataset view, refines results with feedback or an agentic reranker, and imports only new hits. |
| Check | Gate in front of Omniverse NuRec and NCore | Audits raw recordings for calibration, timing, trajectory, and actor defects before any GPU time is spent. |
| Generate | Cosmos Data Factory (Cosmos Transfer, Cosmos Reason) | Submits generation from a recipe, gates every output, and stores each synthetic clip with its source, recipe, and model, plus the judge for every run. |
| Reconstruct | Omniverse NuRec, NCore | Converts drives, launches and tracks runs stage by stage, and returns renders, a 3D view with mesh and splat, and the USDZ scene. |
| Evaluate | Voxel51 platform | Combines image metrics, actor-level checks, and agent review, and routes each run to accepted, retrain, or rejected. |
Each stage runs as a delegated operation in the Voxel51 platform, so GPU work runs on the customer's own compute. For the VW Group, that compute is Azure.
Find: Semantic search over video, where the data already lives
Video is where most curation tools give out: image embedding models don't understand motion, and per-frame labels miss the action. We integrated Cosmos Dataset Search (CDS) as a similarity backend in the Voxel51 platform. Clips are embedded with Cosmos-Embed1 on the CDS side and stored in Milvus, with GPU indexing through cuVS CAGRA available. Text queries are embedded server-side too, so the Voxel51 client needs no GPU and no model download.
In practice, if you type "ego vehicle overtakes a truck towing a trailer" and the results become the sample grid, ready to filter alongside labels, metadata, and model predictions. From there, an engineer can mark good and bad results to refine the query, or hand the top results to an agentic reranker: a planner model writes a rubric from the intent, Cosmos Reason checks each clip against it, and the list is re-ranked with per-clip evidence the engineer can read. Search runs in two directions: within the current dataset, or across the full CDS collection, previewing hits and importing only new ones. Results can be written back as temporal segments, so the unit you hand downstream is the eight seconds that matter, not the hour-long log.
We're benchmarking this on public driving data: 52 driving-scenario queries over 10-second front-camera clips from 1,000 Waymo scenes, comparing image-embedding search (CLIP) on a single keyframe, Cosmos-Embed1 video embeddings in Cosmos Dataset Search, and Cosmos-Embed1 with the agentic reranker on top. We'll publish the precision results in a follow-up.
Check: Don't spend GPU hours on a broken drive
Reconstruction is unforgiving. A camera extrinsic that's off by a few degrees, a LiDAR clock offset, or cuboids sitting a meter underground won't stop a NuRec run. They produce a reconstruction that scores fine and looks wrong, hours later.
So before a drive goes anywhere near a GPU, we audit the raw recording: 80 checks across streams, timing, calibration, trajectory, actors, coordinate frames, and conditioning. It reads MCAP directly, with no conversion, rig file, or cluster required, and it runs in three tiers. The manifest tier takes about a millisecond per drive and is meant for sweeping a whole fleet. The episode tier makes one pass over the file in about a second, and the decoded tier reads the sensor payloads in a few minutes per drive. We built it around one rule: silence is not a pass. Every check reports what it verified and what it couldn't, and links to a page that explains what it measured and how to verify it. Most name the fix and where it lands.
We validated the checks against seven public datasets, read from raw: nuScenes, Waymo, PandaSet, Argoverse 2, MAN TruckScenes, aiMotive, and Lyft Level 5. A check that fires on 100% of a dataset is almost always a bug in the check, not in the data. An Argoverse 2 drive looked like it paired only 31% of its camera frames with LiDAR, because its cameras run at about three times the LiDAR rate. Measured against the pairings actually available, it made 99 of 100.
Across those seven datasets, the audit found 25 issues in the published data, each confirmed by a person. We fixed 17 in our conversion before reconstruction, so every later run gets the fix. The other 8 still reconstruct: the audit names the issue on the drive, and where a fix exists, the Fixes tab applies it to the rig for the next conversion. Measuring what a fix buys takes discipline, because the fix has to be the only thing that changed. On one Waymo drive, correcting timestamps our own converter had set about 50 ms early took the reconstruction from 30.76 to 31.47 dB, with the side cameras gaining the most. On a nuScenes drive the audit flagged, the unfixed run scored 24.77 dB. After applying the audit's fixes, the same recipe reached 27.29 dB.
Across those seven datasets, the audit found 25 issues in the published data, each confirmed by a person. We fixed 17 in our conversion before reconstruction, so every later run gets the fix. The other 8 still reconstruct: the audit names the issue on the drive, and where a fix exists, the Fixes tab applies it to the rig for the next conversion. Measuring what a fix buys takes discipline, because the fix has to be the only thing that changed. On one Waymo drive, correcting timestamps our own converter had set about 50 ms early took the reconstruction from 30.76 to 31.47 dB, with the side cameras gaining the most. On a nuScenes drive the audit flagged, the unfixed run scored 24.77 dB. After applying the audit's fixes, the same recipe reached 27.29 dB.
Generate: Every synthetic clip knows where it came from
Once a scenario family is defined, Cosmos Data Factory expands it with weather, time-of-day, and road-surface variations. From the Voxel51 platform, an engineer writes a recipe and submits the run. Every generated clip comes back as a sample linked to the real clip it came from, carrying the recipe and the model with its settings, and each run records a fingerprint of the judge that scored it. That record is what lets a team improve its synthetic data over time instead of regenerating and hoping.
Each output passes three gates: structural, semantic (Cosmos Reason answers a fixed set of questions about the clip), and attribute (did it come out wet, at night, or foggy, as requested?). The gates store their evidence, every answer and every metric with the threshold it was compared against, so dropping a question or moving a threshold means re-scoring stored results rather than re-running GPUs. On nuScenes, that record changed our decisions in three ways:
- It showed where the money was going. The attribute gate passed only 30 to 38% of clips, so about two-thirds of generation spend went to clips that didn't render the requested condition. Changing the step count didn't move that number, which points the next round of effort at prompts and control settings.
- It showed what to stop paying for. 12 sampling steps matched 35 on accept rate, cutting cost from 6.80 to 2.43 GPU-minutes per clip.
- It showed when to doubt the judge. Three counting questions flagged almost every clip and contradicted themselves across repeat looks. Dropping them, while allowing zero flags on the remaining seven, moved the semantic pass rate from 3% to 91%, and because the evidence was stored, the recalibration took no GPU time.
The same record keeps open questions visible. So far, "icy" has never been confirmed in any of our runs, while "wet" often is. Either the generator can't render ice, or the judge can't tell ice from wet, and the two answers call for different fixes. Every verdict links to its clip and its evidence, so a few minutes in the grid settles it before any fine-tuning effort goes in the wrong direction.
This is how trust in a world model builds up. Each generator version or prompt change can be measured against the same source clips and the same fingerprinted judge, and its results sit next to every earlier run. We're using this setup now to tune Cosmos 3 against Cosmos-Transfer2.5 on the same clips, with the same judge and the same gates.
Reconstruct: NuRec neural reconstruction you can trust
Neural reconstruction is hard to get right. By hand, a single drive through Omniverse NuRec can take days of expert iteration, and a bad calibration or a wrong setting usually surfaces hours into a GPU job. We built the NuRec integration so a team can trust the result without needing a reconstruction expert on every run.
In the Voxel51 platform, an engineer picks an audited drive, converts it to an NCore store, runs a readiness check, and launches. The settings that matter are in the launch form. Compare puts two runs of one drive, or two drives, side by side, and names what differs (drive, recipe, image, budget, split) before it shows the delta, so tuning a drive becomes an experiment you can measure.
Runs take hours, and you can see every step of them. The panel tracks each stage of the job, from conversion and auxiliary data through training to render, validation, mesh, and metrics, and separates time spent waiting for a GPU from time spent working. Without that split, a capacity problem looks like a broken pipeline.
When a run lands, it comes back as a dataset with the original camera beside the render, plus a 3D view of the mesh and the splat. The USDZ scene NuRec produces comes with it, ready for the simulator your team uses, NVIDIA AlpaSim among them, for closed-loop testing of driving models like Alpamayo.
The same workflow covers one drive or many. Engineers can iterate on a single drive, or submit batches to Kubernetes or NVIDIA OSMO and review them together. On 25 public nuScenes scenes on H100s, median peak signal-to-noise ratio (PSNR) was 29.05 dB, and the weakest classes were consistently small or moving ones, led by pedestrians and riders. A pattern like that shows where reconstruction needs work before those scenes reach a simulator.
Evaluate: One number can lie
A reconstruction report usually leads with PSNR, and a single number is easy to fool. One night scene in an earlier nuScenes batch scored 28.59 dB, a number that passes a quick look. Its structural similarity (SSIM) was 0.56, and its learned perceptual image patch similarity (LPIPS) was 0.61, and the renders were visibly poor. PSNR rewards matching large dark regions, and a night scene is mostly dark regions. The panel catches this case directly: when PSNR reads good but SSIM or LPIPS don't, the review card flags the disagreement and the perceptual numbers win.
So no single measurement decides whether a reconstruction is good. We read it in layers:
- Image metrics, split the right way: classical metrics (PSNR, SSIM) and a learned perceptual one (LPIPS), broken out per camera first, then per class. Per camera separates a training problem from one bad camera. Per class shows what the scene is made of: across our 25-scene batch, the weakest classes were consistently small or moving ones, led by pedestrians and riders.
- A separate pass on the actors: a perceptual score over crops of tracked objects sits beside the headline instead of being averaged into it, so a clean road can't hide a smeared pedestrian.
- Detection models, next: we're adding detector-agreement checks that run a detector on both the original and the render, per camera and per class, to test whether a perception stack would see the same world. A scene-level average can hide one camera where the detector finds almost nothing.
- An agent in the loop: Voxel51's NuRec agent skills read the stored metrics, rank a batch weakest-first, check whether the metrics agree with each other, and state the limits that no amount of training removes. Engineers start with the drives that need them.
None of this replaces a human decision. It routes one. Every run lands in a review queue, and the reviewer records an outcome: accepted into the set that feeds simulation and training, sent back for a retrain with the one setting to change, or rejected. Runs where a check couldn't measure anything go to the queue too, because "not measured" is not a pass. Each decision stays attached to the drive, its audit, and the scenario family it came from. The accepted set grows with a record of why every scene is in it, and the rejects show what to look for next.
At Porsche and the Volkswagen Group
Porsche's first goal is virtual validation of driving-software releases: render reconstructed camera inputs from real drives, compare simulated outputs with real-world measurements, and cut reliance on repeated physical test drives. The Voxel51 platform provides the data layer for that workflow on Azure: search and retrieval with Cosmos Dataset Search, data-quality validation before reconstruction, NuRec runs launched and reviewed in the platform, and Cosmos generation with gated outputs. The Volkswagen Group expects the flywheel to reduce repeated physical test-drive effort by up to 90% and shorten development cycles by up to 85%.
What carries over
None of this is specific to one automaker:
- Make the scenario the unit of work, not the log file.
- Gate before you spend. Sweeping a fleet at a millisecond per drive is cheaper than one failed reconstruction.
- Keep provenance on everything you generate, so each model or prompt change is measured against the last one.
- Never trust one number. Break it out by camera, by class, and by the actors in the scene.
- Let a person make the final call, with the evidence attached.
Simulation and synthetic data keep getting cheaper to produce. Knowing which scenario to simulate next, and whether the result is any good, is the part that still needs a data layer.
Try the Voxel51 platform at app.voxel51.com, or explore Cosmos Data Factory on GitHub and deploy it on Azure.
Questions? We have answers.
Audit the raw recording before it gets GPU time. The Voxel51 audit runs 80 checks across streams, timing, calibration, trajectory, actors, coordinate frames, and conditioning, and it reads MCAP directly. The manifest tier takes about 0.2 seconds per drive, fast enough to sweep a whole fleet.
PSNR rewards matching large dark regions, so it can pass a reconstruction that looks wrong. One nuScenes night scene scored 28.59 dB PSNR but 0.56 SSIM and 0.61 LPIPS, and its renders were visibly poor. Break metrics out by camera and by class, and score tracked actors separately.
Cosmos Dataset Search runs as a similarity backend in the Voxel51 platform. Clips are embedded with Cosmos-Embed1 and stored in Milvus on the CDS side, and text queries are embedded server-side, so the client needs no GPU. Results fill the sample grid and can be saved as temporal segments.
Store each generated clip as a sample linked to its source clip, its recipe, its model and settings, and a fingerprint of the judge that scored it. Gate evidence is stored too, so you can re-score results without re-running GPUs.
Head of Physical AI Partnerships
Building the future of Physical AI at Voxel51 🚀
See all articles by Dan Gural
Try Voxel51 for yourself
Loading related posts...
© 2026 Voxel51 All Rights Reserved
