LeRobot Datasets for VLA Training: 5 Real and Synthetic Sets

Q: Which of these synthetic datasets has actual sim-to-real evidence?

Only InternData-A1 — its paper reports zero-shot sim-to-real transfer on several tasks after pretraining a policy exclusively on the synthetic corpus. robocasa-MG_100's paper reports simulation-only BC-Transformer results, and robotwin_unified's paper explicitly cautions against treating its benchmark performance as a real-world deployment proxy without separate validation.

Q: Can synthetic data actually replace real teleoperation data?

The evidence is suggestive but not settled. InternData-A1 reports zero-shot transfer on one task suite, and the result has not been independently reproduced. The SmolVLA result also suggests that data quality and diversity can matter more than raw scale. Treat each sim-to-real claim as one data point, not a general result.

Q: Which licenses permit commercial use?

Apache-2.0 (robocasa-MG_100, robotwin_unified) permits commercial use subject to its terms. CC-BY-4.0 (fmb_multi) permits commercial use with attribution. CC-BY-NC-SA-4.0 (InternData-A1) prohibits commercial use. rh20t_cfg1's 28-episode export mixes CC-BY-SA-4.0 and CC-BY-NC-4.0, so check the per-episode license table before using it commercially.

A field guide to 5 manipulation datasets in LeRobot v3.0, 2 real and 3 synthetic, with their exact fields and three strategies for generating robot data when real teleoperation can't scale.

Large language models train on trillions of tokens. Vision-language models train on billions of images. The largest dataset in this post has 637,498 trajectories, all synthetic. The two real, teleoperated datasets top out at 4,258 and 1,804 episodes. Every real demonstration requires a human at a controller, a physical robot, and a prepared scene. That collection process doesn't scale like web data.

We pulled 5 manipulation datasets into LeRobot v3.0 and FiftyOne: two real, teleoperated single-arm datasets and three simulated datasets spanning single-arm and bimanual setups. This guide examines the data bottleneck, compares three synthetic-generation strategies, and links each dataset to its live FiftyOne Space, paper, and license.

Key takeaways

5 datasets: 2 real teleoperation (fmb_multi on a Franka Panda, rh20t_cfg1 on a Flexiv arm), 3 synthetic (robocasa-MG_100, robotwin_unified, InternData-A1).
Scarcity starts at the source: fmb_multi has 1,804 episodes and rh20t_cfg1 has 4,258. Their FiftyOne exports contain 302 and 28 episodes, respectively. InternData-A1's full synthetic release has 637,498 trajectories, roughly 350× the size of fmb_multi.
The 3 synthetic datasets use different generation strategies: MimicGen decomposes and re-stitches human demonstrations (robocasa-MG_100); an LLM code generator and VLM observer create and repair task programs without seed demonstrations (robotwin_unified); and a compositional pipeline scripts skill policies from an asset library without demonstrations (InternData-A1).
Only the InternData-A1 paper reports sim-to-real transfer: zero-shot transfer after pretraining exclusively on synthetic data. The robocasa-MG_100 and robotwin_unified papers don't make that claim for the exports in this post.
Both real datasets here report fps as nominal, not measured — their source .npy/session data carries no per-frame timestamps, so frames map onto a synthetic grid rather than a recorded rate. Worth knowing before treating fps as ground truth for either.
Licensing: CC-BY-4.0 (fmb_multi), mixed CC-BY-SA-4.0/CC-BY-NC-4.0 by episode (rh20t_cfg1), Apache-2.0 (robocasa-MG_100, robotwin_unified), CC-BY-NC-SA-4.0 (InternData-A1).

Why real robot data is scarce, concretely

Human teleoperation requires trained operators, specialized hardware, real objects, and safety controls.

Architecture can't compensate for narrow data from one robot in one environment. Researchers use simulation, human video, and robot self-exploration to expand the training set beyond what teleoperators can collect.

Scarcity isn't the only problem.

Robot data consists of sequences, and a policy must learn how an action at time T changes the state at T+1. When fps is nominal rather than measured from timestamps, as in both real datasets in this post, the timing is an assumption rather than a recorded fact.

The SmolVLA paper reports state-of-the-art results with an order of magnitude less data than competing approaches and attributes the result to curation rather than collection volume alone. Each synthetic dataset below makes a different choice about how much real data stays in its generation loop.

Three ways to generate synthetic manipulation data

Three synthetic data generation strategies compared by seed data requirement, scale achieved, and published sim-to-real evidence.

Three synthetic data generation strategies compared by seed data requirement, scale achieved, and published sim-to-real evidence.
Dataset	Generation strategy	Seed data	Scale achieved	Sim-to-real evidence
robocasa-MG_100	MimicGen: decompose human demos into object-centric segments, re-stitch into new scene/object configurations, keep only successful rollouts (rejection sampling)	1,250 human teleop demos (4 operators, 3D SpaceMouse)	72,000 trajectories in the full Generated-3000 release; this export is the Generated-100 1/30 subsample, 2,400 episodes	Paper reports BC-Transformer results on the generated data; no sim-to-real transfer claim for this export
robotwin_unified	An LLM code-generation agent writes task programs from natural-language instructions, executes them in a SAPIEN-based simulator, and a VLM observer inspects execution to detect and localize failures for iterative repair	None — fully automated, no human demonstrations at any stage	100,000+ trajectories across 50 tasks and 5 robot embodiments in the source corpus; this export is the full aloha-embodiment slice, 27,500 episodes	Domain randomization (textures, lighting, clutter) is the stated sim-to-real strategy; the source paper cautions against treating benchmark performance as a real-world deployment proxy
InternData-A1	Fully decoupled compositional pipeline: asset specification, scripted skill policies, task composition, and rendering as independent stages — no demonstrations of any kind	None	637,498 trajectories, 401.4M frames, across 4 embodiments and 70 tasks	Paper reports zero-shot sim-to-real transfer on several tasks after pretraining a policy exclusively on this synthetic corpus

MimicGen augments 1,250 real seed demonstrations. RoboTwin needs no seed demonstrations, but its program-synthesis loop uses a VLM observer to detect failures. InternData-A1 also skips demonstrations and composes hand-scripted skill policies across an asset library. Only InternData-A1 reports zero-shot transfer, but one result across three different task suites can't establish which strategy generalizes best. The sim-to-real claim has not been independently reproduced.

The 5 LeRobot datasets at a glance

"Episodes" reflects what's in each live FiftyOne export, not the full upstream source. robocasa-MG_100 and robotwin_unified are complete imports; fmb_multi, rh20t_cfg1, and InternData-A1 are disk-budget-constrained subsets (InternData-A1's export also covers a single task/embodiment out of 1,488 possible task/embodiment directories in its source repo).

The 5 LeRobot v3.0 manipulation datasets in this post, with episode counts as exported to FiftyOne, license terms, and whether the data is real or simulated.

The 5 LeRobot v3.0 manipulation datasets in this post, with episode counts as exported to FiftyOne, license terms, and whether the data is real or simulated.
Dataset	Description	License	Real or Sim
fmb_multi	Contact-rich Franka Panda assembly demonstrations across three boards, phase-labeled by skill primitive	CC-BY-4.0	Real
rh20t_cfg1	Real Flexiv-arm teleoperation with ten synchronized camera views, part of RH20T's one-shot skill-transfer dataset	Mixed CC-BY-SA-4.0 / CC-BY-NC-4.0	Real
robocasa-MG_100	Synthetic kitchen manipulation generated via MimicGen augmentation of real human demonstrations	Apache-2.0	Sim
robotwin_unified	Bimanual ALOHA-embodiment simulation generated by an LLM code-gen agent and VLM observer, with heavy domain randomization	Apache-2.0	Sim
InternData-A1	A single Franka articulation task ("close the laptop") from a fully synthetic, zero-demonstration compositional simulation pipeline	CC-BY-NC-SA-4.0	Sim

The 5 LeRobot datasets in detail

fmb_multi — Contact-rich Franka assembly, phase-labeled

FiftyOne Format Dataset on Hugging Face: Voxel51/fmb-multi-302ep
Try it in a Hugging Face Space: harpreetsahota/fmb-multi-302ep
Original Dataset: robot-lev/fmb_multi (1,804 episodes)
Additional Resources: Paper (arXiv:2401.08553) · Project page
License: CC-BY-4.0

Designed to evaluate generalization in contact-rich assembly tasks, the Functional Manipulation Benchmark focuses on complex insertion-and-regrasp chains where force sensing plays a critical role, rather than basic pick-and-place routines.

fmb_multi serves as its multi-object subset: across three distinct assembly boards, a Franka Panda robot arm performs a structured sequence of skill primitives (approach, grasp, insert, place, regrasp). Throughout these trials, the system collects 6-axis end-effector force/torque readings along with visual and proprioceptive inputs. The benchmark prioritizes cross-object and board adaptability over massive scale, capping the entire suite at under 2,000 trajectories.

In this export, 302 demonstrations are captured (distributed almost evenly across the three boards at 100, 100, and 102 episodes), drawn from a full source collection of 1,804 episodes—showing inherent data sparsity before any FiftyOne filtering.

Available fields:

Camera Streams: 4 video feeds (side_1, side_2, wrist_1, wrist_2; 256×256 AV1 format, converted from original BGR)
Proprioception & State: observation.state (28-dimensional vector covering joint position, velocity, end-effector pose, end-effector velocity, and gripper state), alongside dedicated observation.state.joint_position, .ee_pose, and .gripper fields
Force & Kinematics: observation.force and observation.torque (measured in the end-effector frame), plus a flattened 6×7 observation.jacobian matrix
Control Action: A 7-dimensional Cartesian action vector

rh20t_cfg1 — Flexiv arm, ten camera views, license split by episode

FiftyOne Format Dataset on Hugging Face: Voxel51/rh20t-cfg1-28ep
Try it in a Hugging Face Space: harpreetsahota/rh20t-cfg1-28ep
Original Dataset: robot-lev/rh20t_cfg1 (4,258 episodes; RH20T's full dataset spans 110,000+ sequences across many robot configs, of which cfg1/Flexiv is one)
Additional Resources: Paper (arXiv:2307.00595) · Project page
License: mixed, 16 episodes CC-BY-SA-4.0 / 12 episodes CC-BY-NC-4.0 in this export

Designed to test one-shot skill transfer—specifically whether a robot can master a new task from a single demonstration—RH20T collected an extensive library of over 110,000 manipulation sequences across various robotic platforms and camera setups.

The rh20t_cfg1 subset focuses on a Flexiv arm captured by ten synchronized camera views per episode. It preserves RH20T's scene-based licensing model, dividing scenes roughly equally between permissive and non-commercial terms.

You can verify the scene-to-license mapping for each episode via the folder attribute in meta/rh20t_episodes.json, with the full 28-row breakdown detailed on the dataset card.

Dataset Overview & Key Features

Visual Streams: Includes 10 multi-angle camera feeds (cam_<serial>, 360×640 AV1) recorded from fixed RealSense-style positions.
State & Kinematics: Features a 15-dimensional observation.state vector combining end-effector pose (7-dim), joint positions (7-dim), and gripper status (1-dim). Note that joint values are zeroed out if the source has_joint flag was false.
Force/Torque & Actions: Provides wrist-based observation.force/torque measurements alongside a dedicated 6-axis arm-mounted observation.robot_ft sensor, an 8-dimensional control action (target pose + gripper state), and original meta.rating quality annotations.

robocasa-MG_100 — Full synthetic kitchen dataset via MimicGen

FiftyOne Format Dataset on Hugging Face: Voxel51/robocasa-MG_100
Try it in a Hugging Face Space: harpreetsahota/robocasa-MG_100
Original Dataset: DAVIAN-Robotics/robocasa-MG_100 — all 2,400 episodes, full dataset, no subsetting
Additional Resources: Paper (arXiv:2406.02523) · MimicGen paper (arXiv:2310.17596) · Homepage
License: Apache-2.0

Designed to assess how synthetic data scaling can build upon a limited pool of human demonstrations, RoboCasa features procedurally generated kitchen settings covering 120 distinct configurations across 10 layouts and 12 visual styles.

The MG_100 release aligns with the paper's Generated-100 experimental tier—a 1/30 subset of MimicGen rollouts comprising 24 atomic tasks with 100 trajectories each (totalling 688,750 frames, with episode lengths spanning 89 to 838 frames). MimicGen segments human teleoperation recordings into object-focused primitives, recombines them across varied scene geometries and asset configurations, and validates success through rejection sampling. Because this system multiplies a foundational seed dataset of 1,250 human demonstrations gathered from 4 operators, it relies far more heavily on initial human inputs than the other synthetic benchmarks.

Available Fields:

3 camera streams: 1 wrist-mounted view and 2 static workspace angles (formatted as 128×128 AV1 video).
State Representation: A 16-dimensional observation.state array capturing gripper joint states, base pose, and base-to-end-effector position/quaternion.
Action Space: A 12-dimensional delta-pose action array incorporating base and gripper commands.
Status Flag: A boolean next.done flag.

robotwin_unified — 27,500-episode bimanual simulation with domain randomization

FiftyOne Format Dataset on Hugging Face: Voxel51/robotwin-unified-27500ep
Try it in a Hugging Face Space: harpreetsahota/robotwin-unified-27500ep
Original Dataset: lerobot/robotwin_unified — this export is all 27,500 episodes, full `aloha`-embodiment portion, no subsetting
Additional Resources: Paper (arXiv:2506.18088) · Simulator/generator code (MIT)
License: Apache-2.0

Bypassing human demonstration requirements entirely, RoboTwin 2.0 advances data generation beyond traditional MimicGen-style augmentation.

The process begins with an LLM writing executable task scripts from natural-language instructions, then running them inside a SAPIEN-based simulation environment. A vision-language model evaluates execution frame by frame, pinpointing failures to guide the code-generation agent in self-correcting errors through iterative refinement until high reliability is achieved. robotwin_unified represents the bimanual ALOHA dataset produced by this automated setup: 27,500 episodes comprising 50 dual-arm tasks and 6,075,103 frames. The corpus features extensive domain randomization—spanning 11,000 synthetic background textures, lighting, tabletop elevations, and instruction variants across a 731-object asset library.

With 23,559 unique task strings, the dataset maintains a near 1:1 mapping between instructions and episodes.

Available Fields:

Visual Feeds: 3 camera streams (an overhead cam_high plus dual wrist views cam_left_wrist and cam_right_wrist, 480×640 AV1)
State Representation: A 14-DoF observation.state capturing waist, shoulder, elbow, forearm_roll, wrist_angle, wrist_rotate, and gripper parameters for both left and right arms
Action Space: A 14-dimensional action vector aligned with the state dimensions is not recommended—free-text search and filtering serve as the intended retrieval mechanism.

InternData-A1 — One task, one embodiment, out of 1,488 possible exports

FiftyOne Format Dataset on Hugging Face: Voxel51/interndata-a1-close-the-laptop-369ep
Try it in a Hugging Face Space: harpreetsahota/interndata-a1-close-the-laptop-369ep
Original Dataset: InternRobotics/InternData-A1 (637,498 trajectories, 401.4M frames, 4 embodiments, 70 tasks)
Additional Resources: Paper (arXiv:2511.16651) · Homepage
License: CC-BY-NC-SA-4.0

Among all synthetic datasets highlighted here, InternData-A1 represents the most radical departure from traditional workflows, bypassing human demonstrations entirely—even as an initial seed.

Instead, it relies on a decoupled simulation architecture that synthesizes environment scenes, object assets, and programmatic skill policies. This approach unlocks massive scaling (401.4M frames across 637,498 trajectories) that would be nearly impossible to gather via manual teleoperation. Beyond sheer volume, its most notable milestone is empirical: a VLA policy pretrained solely on this synthetic collection demonstrates zero-shot transfer across multiple physical tasks—making it the only synthetic dataset in this group with published sim-to-real transfer validation.

The upstream repository hosts 1,488 standalone LeRobot v3.0 datasets structured under <embodiment>/<task_category>/<task> directories, each following an independent schema. This specific export isolates franka-1/articulation_tasks/close_the_laptop, featuring 369 out of the 578 episodes available for that task, all guided by the uniform prompt "Close the laptop."

Dataset Features & Structure

Visual Inputs: Two video perspectives (images.rgb.head at 360×640 and images.rgb.hand at 480×640) alongside camera intrinsic and extrinsic parameters.
Kinematics & Control: Granular state and action values, including states.joint.position, actions.gripper.pose, and leader-arm teleoperation signals under master_actions.*.
Derived Convenience Aggregations: Includes a 49-dimensional observation state vector and a 43-dimensional action vector, engineered post-import to align with FiftyOne's State & Action visualizer natively.

Frequently Asked Questions

Every real demonstration requires a human operator, a physical robot, a real environment with real objects, and safety overhead around all of it — none of which parallelizes the way web scraping does. The two real datasets in this post (fmb_multi, rh20t_cfg1) top out at 1,804 and 4,258 episodes at the source; the largest synthetic dataset here (InternData-A1) is 637,498 trajectories, because simulation removes the human-in-the-loop requirement for every single trajectory.

Only InternData-A1 — its paper reports zero-shot sim-to-real transfer on several tasks after pretraining a policy exclusively on the synthetic corpus. robocasa-MG_100's paper reports simulation-only BC-Transformer results, and robotwin_unified's paper explicitly cautions against treating its benchmark performance as a real-world deployment proxy without separate validation.

It means the frame rate in the dataset's metadata wasn't measured from actual per-frame timestamps—it's a value assumed for a synthetic, evenly spaced frame grid. Both fmb_multi (source .npy files carry no timestamps) and rh20t_cfg1 (cameras were downsampled from variable native capture rates) report nominal fps. If your training pipeline assumes fps reflects real elapsed time between frames, that assumption doesn't hold for either dataset as tightly as it does for datasets with measured timestamps.

The evidence is suggestive but not settled. InternData-A1 reports zero-shot transfer on one task suite, and the result has not been independently reproduced. The SmolVLA result also suggests that data quality and diversity can matter more than raw scale. Treat each sim-to-real claim as one data point, not a general result.

Apache-2.0 (robocasa-MG_100, robotwin_unified) permits commercial use subject to its terms. CC-BY-4.0 (fmb_multi) permits commercial use with attribution. CC-BY-NC-SA-4.0 (InternData-A1) prohibits commercial use. rh20t_cfg1's 28-episode export mixes CC-BY-SA-4.0 and CC-BY-NC-4.0, so check the per-episode license table before using it commercially.

5 Real and Synthetic LeRobot Datasets for VLA Training in 2026

Talk to an AI expert

Key takeaways

Why real robot data is scarce, concretely

Three ways to generate synthetic manipulation data

The 5 LeRobot datasets at a glance

The 5 LeRobot datasets in detail

fmb_multi — Contact-rich Franka assembly, phase-labeled

Available fields:

rh20t_cfg1 — Flexiv arm, ten camera views, license split by episode

Dataset Overview & Key Features

robocasa-MG_100 — Full synthetic kitchen dataset via MimicGen

Available Fields:

robotwin_unified — 27,500-episode bimanual simulation with domain randomization

Available Fields:

InternData-A1 — One task, one embodiment, out of 1,488 possible exports

Dataset Features & Structure

Frequently Asked Questions

Talk to an AI expert

Related posts

Related posts

5 Real and Synthetic LeRobot Datasets for VLA Training in 2026

Talk to an AI expert

Key takeaways

Why real robot data is scarce, concretely

Three ways to generate synthetic manipulation data

The 5 LeRobot datasets at a glance

The 5 LeRobot datasets in detail

fmb_multi — Contact-rich Franka assembly, phase-labeled

Available fields:

rh20t_cfg1 — Flexiv arm, ten camera views, license split by episode

Dataset Overview & Key Features

robocasa-MG_100 — Full synthetic kitchen dataset via MimicGen

Available Fields:

robotwin_unified — 27,500-episode bimanual simulation with domain randomization

Available Fields:

InternData-A1 — One task, one embodiment, out of 1,488 possible exports

Dataset Features & Structure

Frequently Asked Questions

Why is real robot data so much scarcer than simulated data?

Which of these synthetic datasets has actual sim-to-real evidence?

What does "fps is nominal" mean and why does it matter?

Can synthetic data actually replace real teleoperation data?

Which licenses permit commercial use?

Talk to an AI expert

Related posts

Related posts