Book a demo
Blog
Physical AI

From Franka Arms to Humanoids: 11 LeRobot v3.0 Datasets for VLA Training in 2026

Sep 16, 2026
•
22 min read
Author
Harpreet Sahota avatar
Harpreet Sahota
Harpreet is Hacker-in-Residence at Voxel51, where he turns cutting-edge AI ideas into open-source prototypes and demos that push the boundaries of deep learning. From building tools that inspire to creating content that educates, Harpreet helps the AI community level up—one wild idea at a time.
See all articles by Harpreet Sahota

Talk to an AI expert

Book a demo
A field guide to 11 robot-learning datasets, from Franka arms to full humanoids and real teleoperation to simulation, reformatted into LeRobotDataset v3.0 and loaded into FiftyOne with their exact fields and per-frame streams.
A vision-language-action (VLA) model doesn't learn from a folder of images.
It learns from episodes: a camera stream, a robot's joint state, and the action it took, all stamped to the same clock and lined up frame by frame. Get that alignment wrong, and the model can't tell which arm motion caused which visual change. The training signal disappears.
This makes data engineering a prerequisite for modeling.
We pulled together 11 datasets in LeRobot v3.0, from a Franka arm to a Unitree G1 humanoid and a fully synthetic manipulation suite. We reformatted or subsetted each dataset and loaded it into FiftyOne, where you can inspect video streams, state/action vectors, and task labels before downloading.
This guide explains the data problem, compares LeRobot with the raw-sensor-log alternative MCAP, breaks down the FiftyOne schema, and links each dataset to its live FiftyOne Space, paper, and license.

Key takeaways

  • 11 datasets, all loadable today with fiftyone.utils.huggingface.load_from_hub, spanning single Franka arms to full humanoids (HIW-500-LeRobot on Unitree G1, tavis-head-gr1t2, tavis-head-reachy2) to a bimanual dexterous-hand platform (trex_dataset on Dexmate Vega-1).
  • 4 of the 11 FiftyOne exports carry the complete source episode set (robocasa-MG_100: 2,400/2,400; robotwin_unified: 27,500/27,500; tavis-head-gr1t2 and tavis-head-reachy2: 800/800 each). The other 7 are disk-budget-constrained subsets of much larger sources — as small as 10 of AgiBotWorld2026's 102,078 episodes and 23 of trex_dataset's 5,464.
  • 6 contain real-world teleoperation or human-collected data; 5 are simulated (InternData-A1, robocasa-MG_100, robotwin_unified, tavis-head-gr1t2, tavis-head-reachy2). InternData-A1's preview clips look photorealistic despite being 100% simulator output.
  • droid_3d's live FiftyOne export is RGB-only — the full source dataset's depth video and point-cloud streams use dtype values (depth_video, pointcloud) that FiftyOne's LeRobot importer doesn't recognize, so they're absent from the FiftyOne dataset even though they exist upstream.
  • Licenses vary, sometimes within one export: MIT (droid_3d, trex_dataset), Apache-2.0 (robocasa-MG_100, robotwin_unified), CC-BY-4.0 (fmb_multi, HIW-500-LeRobot, tavis-head-*), and CC-BY-NC-SA-4.0 (AgiBotWorld2026, InternData-A1). The 28-episode rh20t_cfg1 export mixes CC-BY-SA-4.0 and CC-BY-NC-4.0 by episode.

Why data organization matters more than architecture

Source: arXiv 2505.04769
A vision-language-action (VLA) model fuses three things:
  1. a vision encoder that turns camera frames into tokens
  2. a language encoder that turns instructions into tokens
  3. and an action decoder that turns the fused representation into joint commands or end-effector deltas.
Most VLAs bolt this onto a pretrained vision-language backbone, because CLIP- or SigLIP-scale visual and linguistic priors are cheap to get from the internet.
Action grounding is expensive. A video of a person picking up a mug has no joint angles, end-effector pose, or gripper command attached. A robot policy needs that information to reproduce the motion. Vision-language models (VLMs) can draw on web-scale data; VLAs don't have equivalent action data.
Source: Generated using NanoBanana 2
That gap creates five differences from typical vision-language training data:
  • Temporal structure. Episodes are ordered sequences, not shuffleable independent samples — a policy has to model how actions at time T affect the state at T+1.
  • Proprioception. Joint positions, velocities, and end-effector poses are low-dimensional (10-50 values) but essential — vision shows where the object is, not where the robot's own arm is.
  • Action grounding. Understanding "pick up the cup" doesn't provide the motor command sequence needed to grasp that cup in that configuration.
  • Embodiment heterogeneity. A 7-DoF single arm, a 14-DoF bimanual rig, and a humanoid's 29-DoF whole-body state don't share an action space, so the datasets require remapping before concatenation.
  • Data scarcity. Teleoperation is slow and expensive to collect, so robot datasets run to thousands or millions of trajectories — not the billions of tokens or images that make LLM/VLM pretraining work.
Without a common structure, these differences block dataset combination.
RLDS provides a lossless episode/step hierarchy, Open X-Embodiment coarsely aligns actions into 7-DoF vectors across 22 embodiments, and LeRobot packages synchronized episodes in a compressed format. The 11 datasets below share that organization, which exposes the remaining differences in embodiments, sensors, state, and action.

LeRobot v3.0 vs. MCAP: two different jobs

If you've read our companion piece, 10 Multimodal MCAP Datasets in 2026, you've seen the other half of this problem. MCAP and LeRobot both solve robot-data organization, but for opposite inputs:
How MCAP and LeRobot v3.0 differ across storage, compression, and typical use.
How MCAP and LeRobot v3.0 differ across storage, compression, and typical use.
.MCAPLeRobot v3.0
What it storesContinuous, multi-rate sensor log (camera, LiDAR, IMU, GPS, all on their own clocks)Discrete, synchronized episodes (state/action/vision aligned to one frame grid)
CompressionLossless (LZ4/Zstandard at the chunk level)Lossy video compression (AV1/H.264), trades fidelity for size
Typical useSLAM, sensor fusion, perception benchmarking, raw log archivalImitation learning, VLA pretraining and fine-tuning
One sample isOne continuous recording (minutes long), decoded liveOne episode (seconds to a couple minutes), pre-chunked into fixed frame steps
StreamingChunk-indexed seeking within a fileStreamingLeRobotDataset streams directly from the Hub, no local download
In FiftyOnedataset.media_type == "multimodal", one sample = one continuous recordingdataset.media_type == "multimodal", one sample = one episode, per-frame data resolved through media_reference
If you have 20 minutes of unsynchronized LiDAR, camera, and IMU data and need to seek to timestamp 4:32, use MCAP. If you have labeled robot demonstrations and need synchronized state/action/vision triples for training, use LeRobot. The formats cover different parts of the robot-data pipeline.
Source: Generated using NanoBanana 2
LeRobot v3.0 changes how it packages episodes. Version 2.1 used one Parquet file and one MP4 per episode, which strains filesystem limits at millions of episodes. Version 3.0 packs many episodes into shared Parquet and MP4 files and resolves them through relational metadata. That structure makes a 102,078-episode, 13.6TB release such as AgiBotWorld2026 practical to package.

The FiftyOne LeRobot schema: what's a sample, what's a stream

Every one of the 11 datasets below is loaded into FiftyOne the same way, so the field schema is consistent across all of them. Each is a multimodal FiftyOne dataset where one sample equals one episode. The sample-level fields you can query, filter, and sort on in the App are:
Sample-level fields in a FiftyOne LeRobot dataset, and what each one holds.
Sample-level fields in a FiftyOne LeRobot dataset, and what each one holds.
FieldFiftyOne typeWhat it holds
idObjectIdFieldFiftyOne sample id
media_referenceMediaReferenceFieldPointer into the LeRobot source's data/*.parquet, videos/*/*.mp4, and meta/ files for this episode — resolved on demand, not duplicated per sample
tagsListField(StringField)FiftyOne sample tags (empty by default)
metadataMetadataStandard FiftyOne sample metadata (size_bytes, mime_type)
created_at / last_modified_atDateTimeFieldFiftyOne bookkeeping timestamps
episode_indexIntFieldEpisode index within the export
taskStringFieldPrimary task label for the episode (free text, not fo.Classification — see below)
tasksListField(StringField)Full list of task label(s) for the episode
lengthIntFieldNumber of frames in the episode
durationFloatFieldEpisode duration in seconds (length / fps)
robot_typeStringFieldRobot platform identifier (empty/null for several of these 11 — see per-dataset notes)
fpsFloatFieldRecording frame rate
Why is `task` a string instead of a `Classification`? These datasets store task identity as free text: an instruction, a skill-primitive name, or a placeholder such as "task 1". A fo.Classification field would imply a closed label set with confidence scores. robotwin_unified alone has 23,559 distinct task strings across 27,500 episodes. Use task and tasks for text search and grouping.
Where does per-frame data live? Video streams (observation.images.*), proprioceptive state (observation.state, observation.state.*), actions, and per-frame sensor data stay in the referenced LeRobot data/*.parquet and videos/*/*.mp4 files. FiftyOne surfaces them in the State & Action, Streams, and Statistics tabs as you scrub through an episode. A 60fps humanoid episode can carry a 44-dimensional state vector and 4 video streams per frame, so flattening these values into top-level sample fields would not scale. media_reference avoids that duplication.

The 11 LeRobot datasets at a glance

"Episodes" below reflects what's actually in each live FiftyOne export, not the full upstream source. 4 of 11 are complete imports; the other 7 are disk-budget-constrained subsets (as small as 10-68 episodes) selected for shard contiguity, not statistical representativeness — each entry states the selection method and the full source size for comparison.
The 11 LeRobot v3.0 datasets, with sensors, license, and whether the data is real or simulated.
The 11 LeRobot v3.0 datasets, with sensors, license, and whether the data is real or simulated.
DatasetDescriptionSensorsLicense (this export)Real or Sim
AgiBotWorld2026 Real-world dual-arm humanoid manipulation collected via RL-style human-intervention rollout on the AgiBot G2 platformRGB + depth, 7 cams, RL signalsCC-BY-NC-SA-4.0Real
droid_3d RGB-only slice of a large real-world DROID manipulation dataset whose full release adds depth and point clouds for 3D-aware policiesRGB only, 3 camsMITReal
fmb_multi Contact-rich Franka Panda assembly demonstrations across three boards, phase-labeled by skill primitiveRGB + force/torque, 4 camsCC-BY-4.0Real
HIW-500-LeRobot Whole-body Unitree G1 humanoid teleoperation performing household chores across real homes in Southeast AsiaRGB stereo, 3 camsCC-BY-4.0Real
InternData-A1 A single Franka articulation task ("close the laptop") from a fully synthetic, zero-demonstration compositional simulation pipelineRGB, 2 camsCC-BY-NC-SA-4.0Sim
rh20t_cfg1 Real Flexiv-arm teleoperation with ten synchronized camera views, part of RH20T's one-shot skill-transfer datasetRGB, 10 camsMixed CC-BY-SA-4.0 / CC-BY-NC-4.0Real
robocasa-MG_100 Synthetic kitchen manipulation generated via MimicGen augmentation of real human demonstrationsRGB, 3 camsApache-2.0Sim
robotwin_unified Bimanual ALOHA-embodiment simulation generated by an LLM code-gen agent and VLM observer, with heavy domain randomizationRGB, 3 camsApache-2.0Sim
tavis-head-gr1t2 Simulated active-vision humanoid benchmark (GR1T2) comparing head-mounted vs. fixed cameras on identical manipulation tasksRGB, active-vision head + fixed + 2 wrist camsCC-BY-4.0Sim
tavis-head-reachy2 The same active-vision benchmark as tavis-head-gr1t2, run on a Reachy2 humanoid insteadRGB, active-vision head + fixed + 2 wrist camsCC-BY-4.0Sim
trex_dataset Bimanual dexterous manipulation with real per-fingertip tactile sensing on a Dexmate Vega-1 robotRGB + tactile, 3 RGB + 20 tactile streamsMITReal

The 11 LeRobot datasets in detail

AgiBotWorld2026: real-world dual-arm manipulation with RL collection signals

AgiBot World is one of the largest real-world humanoid manipulation datasets released to date.
A team collected long-horizon household and commercial tasks using the AgiBot G2 dual-arm platform. They deployed it in real homes, retail spaces, and offices—not just a single controlled lab.
The 2026 release has three collection modes:
  • Pure imitation learning
  • RL-style rollout with human intervention (HG-DAgger)
  • Richer multi-signal interaction logging
And three annotation layers on top of the raw episodes:
  • Subtask segmentation
  • 2D bounding boxes
  • Step-level language instructions
This FiftyOne export pulls from the HG-DAgger reinforcement-learning slice specifically, so you're looking at a robot mid-training, complete with reward signals and human-correction flags, not a polished expert demonstration. All 10 episodes come from a single ReinforcementLearning/Home/task_12192/HG-DAgger archive, one task ("Insert the loose change into the coin slot of the piggy bank dagger").
Because this is the RL collection category, every frame carries collection-process signals most imitation-learning datasets don't: reward, done, succeed, intervened (whether a human corrected the policy at that frame), truncated, and action_prob.
Fields you get: 7 camera streams (top_head, hand_left, hand_right, two head fisheyes, a rear fisheye, and a head_depth stream — declared dtype: video with gray16be pixel format, so it imports like RGB despite being depth data), a 216-dim observation.state vector, and a 44-dim action vector.
Known quirks. The 12.3TB source repo isn't a standard flat LeRobot layout — it ships as per-task .tar.gz archives under ImitationLearning/, ReinforcementLearning/, and RichInteraction/, and these three categories declare incompatible observation.state shapes (193 / 216 / 169 respectively), so a single LeRobot v3 export can only cover one category at a time. This export stays inside ReinforcementLearning. The source ships LeRobot v2.1 and was converted with lerobot.scripts.convert_dataset_v21_to_v30; AgiBot's three custom annotation layers (key_frame, instruction_segments, subtask/bounding-box labels) are dropped by that converter and aren't present here.

HIW-500-LeRobot — Humanoid teleoperation in real homes, sampled across all 11 tasks

Humanoids In-the-Wild is exactly what it sounds like.
The dataset captures hundreds of hours showing a Unitree G1 humanoid handling domestic tasks throughout actual residences in Southeast Asia. Environments, illumination, object arrangements, and individual operator control patterns naturally fluctuate between demonstrations—by design, offering raw, uncurated field recordings rather than sanitized laboratory trials. This curated selection spans all 11 task classes from the source collection (ranging from table preparation to hanging keys on pegs), delivering a representative window into the complete 23,743-trajectory dataset without requiring a multi-terabyte retrieval.
Rather than extracting the minimal contiguous block (which would limit coverage to a single activity), we sampled 48 demonstrations across 6 distinct file groups to ensure representation across every task class—from "setting the table" (15 demonstrations) to "hang hanger" (a single demonstration).
Fields you get: 3 camera streams (head at 480×1280, left_wrist/right_wrist at 480×640, all AV1), a 29-DoF observation.state (hips, knees, ankles, waist, shoulders, elbows, wrists), a 23-dim observation.state.wbc whole-body-control vector (pivot velocity/pose + both end-effector poses + trigger/squeeze), and a matching 23-dim action.
Known quirks. The source's language_persistent and language_events fields carried the dataset's 161 fine-grained subtask labels and 148K+ annotations; this export includes only the 11 coarse task categories, not the fine-grained subtask text. Camera intrinsics/extrinsics mentioned in the source docs also aren't exposed as FiftyOne fields.

tavis-head-gr1t2 and tavis-head-reachy2 — Active-vision humanoids, full datasets, in simulation

TAVIS asks a specific, narrow question that most robot-learning datasets don't bother to isolate: does letting a humanoid move its own head/camera during manipulation actually help, and can you measure that benefit directly?
The benchmark pairs identical tasks and identical camera rigs across two different humanoid torsos — Fourier's GR1T2 and Pollen's Reachy2 — and records both a head-mounted "active vision" stream and a fixed-camera view for every episode, so you can train and compare policies with the head-camera on or off using the same demonstrations. It's simulation-only (built in NVIDIA IsaacLab), VR-teleoperated, and small (800 episodes per robot) by design, since the goal is controlled comparison, not scale.
Both robots share 5 task classes: ClutterPickCubeTask, ClutterPickLiftTask, ConditionalPickTask, MultiShelfScanTask, WaitThenActTask (250/250/100/100/100 episode split).
Fields you get: 4 camera streams (OBS_HEAD, OBS_FIXED, OBS_WRIST_LEFT, OBS_WRIST_RIGHT, all 480×640 AV1 at 60fps), a 19-dim action (bimanual arm targets + neck + grippers), observation.state (44-dim for GR1T2, 36-dim for Reachy2 — the two robots have different DoF counts, so don't assume the vectors are interchangeable), separate left/right end-effector position and orientation fields, and a per-frame language_instruction string (33 distinct values for Reachy2).
Known quirks. robot_type is null in both sources despite the org page naming the robots explicitly. A separate tavis-hands-* pair (300 episodes each, testing local-occlusion perception via wrist cameras rather than head search) exists in the same org but isn't part of this roundup.

trex_dataset — Bimanual dexterous manipulation with fingertip tactile sensing

T-Rex is built around a sensing modality almost no other manipulation dataset bothers with at this density: real per-fingertip tactile sensing.
This dataset captures five sensors per hand, on a bimanual Dexmate Vega-1 robot fitted with Sharpa Wave dexterous hands. Each episode includes not just RGB video but 20 additional tactile video streams (a raw sensor image and a deformation map per fingertip), plus 6-axis force/torque readings at each fingertip, collected via Manus glove and VIVE tracker teleoperation across 207 objects and 22 distinct motor primitives.
If you care about contact-rich, touch-conditioned manipulation rather than vision-only policies, this is one of the only public datasets built specifically to support it.
The 23-episode export here is the first contiguous shard of the full 5,464-episode dataset.
Fields you get: 3 RGB camera streams (head_left, left_wrist, right_wrist, 640×360) plus 20 tactile video streams — a raw sensor image and an estimated deformation map per fingertip, 2 hands × 5 fingers × 2 stream types — a 58-dim observation.state and matching action ([L_arm 7 | L_hand 22 | R_arm 7 | R_hand 22] joint layout), and a 60-dim observation.tactile_force (per-fingertip 6-axis wrench).
Known quirks. The 20 tactile streams are encoded losslessly (libx264 -qp 0, since pixel values are physically meaningful sensor readings), which forces the H.264 High 4:4:4 Predictive profile — most browsers can't decode this in a plain <video> element. As a result, these streams show no thumbnail in a generic viewer. FiftyOne, ffmpeg, PyAV, or torchcodec decode them fine; the 3 RGB streams use standard yuv420p and preview everywhere.

fmb_multi — Contact-rich Franka assembly, phase-labeled

The Functional Manipulation Benchmark was built to stress-test generalization in contact-rich assembly.
Moving beyond basic pick-and-place actions, this benchmark evaluates complex insertion and regrasping sequences where force feedback plays a critical role. The fmb_multi subset focuses on multi-object manipulation across three distinct assembly boards. Each trajectory features a Franka Panda robotic arm executing a structured skill workflow—approach, grasp, insert, place, and regrasp—while capturing 6-axis end-effector force/torque dynamics, along with visual and proprioceptive signals. Rather than aiming for sheer volume, the benchmark remains intentionally compact (under 2,000 total episodes) to prioritize testing policy generalization across diverse objects and board configurations.
The exported 302-episode subset is distributed almost uniformly across the three boards, containing 100, 100, and 102 episodes, respectively.
Fields you get: 4 camera streams (side_1, side_2, wrist_1, wrist_2, 256×256 AV1, converted from source BGR), observation.state (28-dim: joint position + velocity + ee_pose + ee velocity + gripper), separate observation.state.joint_position/.ee_pose/.gripper fields, observation.force/observation.torque (end-effector frame), a flattened 6×7 observation.jacobian, and a 7-dim cartesian action.
Known quirks. task here is the skill primitive active at frame 0 only — not an episode summary. Use tasks for the full ordered sequence an episode passes through. Depth was dropped by the upstream port (not this export); the source FMB release includes 4 depth maps per frame that never made it into fmb_multi. fps: 10 is nominal — the original .npy files carry no real timestamps, so frames map 1:1 onto a synthetic grid rather than a measured rate. Per-episode board/object_id metadata lives in the source's meta/fmb_episodes.json, not as a FiftyOne field — join back to it by episode_index if you need it.

rh20t_cfg1 — Flexiv arm, ten camera views, license split by episode

RH20T tests one-shot skill transfer: can a robot learn a new task from just a single demonstration?
To answer this, it collects an enormous spread of over 110,000 manipulation sequences across diverse robot platforms and camera setups. The rh20t_cfg1 subset focuses on a single Flexiv arm, paired with ten synchronized camera views per episode. RH20T licenses its data per scene, and this port preserves that original split rather than applying a blanket license. Roughly half the scenes permit commercial use, while the rest are non-commercial only.
Check meta/rh20t_episodes.json's folder field to identify the scene and licensing terms for each episode.
Fields you get: 10 camera streams (cam_<serial>, 360×640 AV1, one per fixed RealSense-style camera), observation.state (15-dim: ee_pose 7 + joint 7 + gripper 1 — note joint is all-zero for episodes where the source's has_joint flag was false), observation.force/torque plus a separate robot-mounted observation.robot_ft (6-axis), an 8-dim action (next ee_pose + gripper target), and a per-episode meta.rating quality score from the original RH20T annotation pass.
Known quirks. task values here are literally "task 1" through "task 6" — the LeRobot port never carried over RH20T's original natural-language task descriptions, only a numeric cluster id. Treat task as an opaque grouping key, not a description. Audio sidecars from the source (one .wav per episode) are deliberately excluded — they sit outside the declared LeRobot schema entirely, and the source data includes volunteer-recorded human-robot interaction with faces/voices that warrants care regardless.

robocasa-MG_100 — Full synthetic kitchen dataset via MimicGen

RoboCasa constructs 120 fully procedurally generated kitchen environments across ten distinct layouts and a dozen aesthetic styles to test how far synthetic augmentation can stretch a limited collection of human demonstrations.
The MG_100 dataset shows the MimicGen pipeline in action: rather than logging thousands of additional manual rollouts, researchers broke down 1,250 human teleoperation demonstrations into modular, object-centric primitives. They then synthesized 100 new rollouts for every task by stitching those components back together inside unseen spatial setups, keeping only successful executions. While synthetic, this approach directly addresses a key robotics bottleneck: evaluating how effectively automated generation can replace labour-intensive physical data collection. Specifically, MG_100 mirrors the paper's Generated-100 setup, taking a 1/30 random slice from the broader MimicGen output to yield 100 demonstrations across 24 atomic tasks (spanning 688,750 total frames, with trajectory lengths between 89 and 838 frames).
Fields you get: 3 camera streams (2 static workspace views + 1 wrist-mounted eye-in-hand, all 128×128 AV1), a 16-dim observation.state (base-to-eef position/quaternion + gripper qpos + base position/quaternion), a 12-dim delta-pose+gripper+base action, and a next.done termination flag.
Known quirks. Task episode counts are imbalanced by design — fixed-object tasks (e.g. "turn on the sink faucet") have exactly 100 episodes, while pick-and-place tasks split across many object-category variants, as few as 1 episode for some combinations, because MimicGen samples object identity per generation attempt. NavigateKitchen, the paper's 25th atomic task, is absent from every MimicGen-generated release (including this one) because MimicGen can't synthesize mobile-navigation trajectories — that's a property of the source data, not something excluded on import.

robotwin_unified — 27,500-episode bimanual simulation with domain randomization

RoboTwin 2.0 takes data generation a step beyond MimicGen-style augmentation—it removes the human demonstration requirement entirely.
An LLM writes executable task programs from a natural-language instruction, a SAPIEN-based simulator runs them, and a vision-language model watches the execution frame-by-frame to catch failures and drive the code-generation agent to fix its own mistakes, iterating until the task succeeds reliably. robotwin_unified is the bimanual ALOHA slice of that pipeline's output: 50 dual-arm tasks, heavy domain randomization across textures, lighting, and clutter, and a task-instruction vocabulary so varied that nearly every one of its 27,500 episodes has a unique description (23,559 distinct task strings, near one-to-one).
Domain randomization spans scene clutter (731-object asset library), background textures (11,000 AI-generated textures), lighting, tabletop height, and instruction phrasing.
Fields you get: 3 camera streams (cam_high overhead + cam_left_wrist/cam_right_wrist, 480×640 AV1), a 14-DoF observation.state (left/right waist, shoulder, elbow, forearm_roll, wrist_angle, wrist_rotate, gripper), and a matching 14-dim action.
Known quirks. Entirely simulated (SAPIEN-based) — the source paper itself cautions against treating performance here as a proxy for real-world deployment readiness without separate sim-to-real validation. Because near every episode has a unique task string, don't try to build a Classification schema over task — text search/filtering is the right access pattern here, not label grouping.

droid_3d — RGB-only in FiftyOne; depth and point clouds need a different loader

DROID is one of the most widely used real-robot manipulation datasets in the field.
This dataset is assembled by pooling data collection across many labs on a standardized rig, and droid_3d builds on that foundation by adding a genuinely useful extension: synchronized depth video and quantized point clouds alongside the original RGB, purpose-built to support 3D-aware manipulation and representation-learning research (EmbodiedMAE). The catch for anyone working purely in FiftyOne or stock LeRobot is that those depth and point-cloud streams use dtypes neither tool currently decodes, so you get the RGB-and-language backbone of a dataset whose real distinguishing feature lives just outside this export.
The importer recognizes video and image dtypes, while the source declares depth and point clouds as depth_video and pointcloud, which have no built-in FiftyOne import path.
Fields you get: 3 RGB camera streams (wrist, external_0, external_1, 224×398 AV1), an 8-dim action, and up to 3 independent natural-language task descriptions per episode (language_1/_2/_3 — different phrasings/emphasis of the same action, not paraphrases; only language_1 is surfaced as the sample-level task).
Known quirks. If you need the depth or point-cloud streams, this FiftyOne export won't have them at any episode count — install lerobotdataset3d (pip install git+https://github.com/ZibinDong/lerobotdataset3d.git) and load the full ZibinDong/droid_3d repo directly with LeRobotDatasetDepthPointcloud instead. robot_type is empty here because DROID spans multiple robot platforms, with no single embodiment tag recorded at the dataset level.

InternData-A1 — One task, one embodiment, out of 1,488 possible exports

InternData-A1 pushes the synthetic-data bet furthest of any dataset in this series: no human demonstrations anywhere in its pipeline, not even as a seed.
Instead, a fully decoupled simulation stack composes scenes, objects, and scripted skill policies from an asset library. It renders the results at a scale (637,498 trajectories, 401.4M frames) that would be genuinely difficult to reach through teleoperation. The paper's headline result is what makes it worth paying attention to beyond the scale: a policy pretrained exclusively on this synthetic data reportedly transfers zero-shot to several real-world tasks, which is real (if early) evidence that scale and diversity in simulation can substitute for at least some real-world collection. The source repo contains 1,488 independent LeRobot v3.0 datasets, one per <embodiment>/<task_category>/<task> directory, each with its own schema.
This export covers franka-1/articulation_tasks/close_the_laptop: 369 of that task's 578 episodes, the largest complete shard-0 set obtainable without downloading every shard. Every episode shares the instruction "Close the laptop."
Fields you get: 2 camera streams (images.rgb.head 360×640, images.rgb.hand 480×640), camera intrinsics/extrinsics vectors, and a long list of per-component state/action fields (states.joint.position, actions.gripper.pose, master_actions.* for the teleop leader-arm commands, etc.) — plus two derived convenience fields, a 49-dim observation.state and 43-dim action, packed post-import specifically so FiftyOne's State & Action viewer tile (which only renders features literally named observation.state/action) can display this dataset's proprioception at all.
Known quirks. 100% synthetic — generated through a compositional simulation pipeline (asset specification, skill policies, task composition, and rendering as decoupled stages), not collected on real hardware, despite photorealistic preview clips. franka-1 and franka-2 are two *different* feature spaces in the source (differing image shape and gripper value range) — this export uses only franka-1; don't merge episodes across the two without checking meta/info.json on each. Because the source's 1,488 task directories span embodiments with genuinely incompatible schemas (single-arm Franka vs. dual-arm Genie-1/Lift-2/Split-Aloha), this export covers exactly one task/embodiment pair rather than a cross-embodiment sample.

Frequently Asked Questions

Harpreet Sahota avatar
Harpreet Sahota
Harpreet is Hacker-in-Residence at Voxel51, where he turns cutting-edge AI ideas into open-source prototypes and demos that push the boundaries of deep learning. From building tools that inspire to creating content that educates, Harpreet helps the AI community level up—one wild idea at a time.
See all articles by Harpreet Sahota

Talk to an AI expert

Book a demo
Loading related posts...
Talk to an AI expert
Product
Data SearchData CurationData AnnotationModel EvaluationMultimodalData GenerationIntegrationsPluginsPricingSecurity
Solutions
AgricultureAutonomous SystemsDefenseHealthcareManufacturingRetailRoboticsSecurity
Developers
DocumentationEvents & MeetupsVisual and Physical AI GlossaryCommunityVoxel51 Agent
Resources
BlogOn-Demand WebinarsCustomer StoriesModel ZooDataset ZooCompareResearch
Company
About Voxel51CareersPress
© 2026 Voxel51 All Rights Reserved
Terms of Service
Privacy Policy