Author
Harpreet Sahota
Harpreet is Hacker-in-Residence at Voxel51, where he turns cutting-edge AI ideas into open-source prototypes and demos that push the boundaries of deep learning. From building tools that inspire to creating content that educates, Harpreet helps the AI community level up—one wild idea at a time.
See all articles by Harpreet Sahota
In this article
Talk to an AI expert
A field guide to 5 humanoid and bimanual manipulation datasets in LeRobot v3.0 — with the exact state/action dimensionality, fields, and research behind each one, and why "humanoid dataset" hides a 9x spread in how these robots actually represent themselves.
Two robots doing near-identical tasks—reach into clutter, pick something up, place it somewhere—can describe that action with a 19-dimensional vector or a 216-dimensional one, depending entirely on how many joints, cameras, and collection-process signals the platform records. Neither number is wrong. They're just different robots, and a policy trained on one has no native way to use data from the other.
That's embodiment heterogeneity, one of the five reasons robot data is fundamentally harder to organize than the image and text data that trains typical vision-language models.
We pulled 5 humanoid and bimanual manipulation datasets — real teleoperation and pure simulation, a Unitree G1 to a pair of matched humanoid torsos — into LeRobot v3.0 and FiftyOne, and the spread in how they represent the same broad category of task is the throughline of this post. Below: what embodiment heterogeneity actually costs in practice, a side-by-side of state/action dimensionality across all 5, and a tour of each with links to the live FiftyOne Space, the paper, and the license.
Key takeaways
- 5 datasets spanning a dual-arm mobile manipulator (
AgiBotWorld2026), a full humanoid on legs (HIW-500-LeRobot, Unitree G1), two matched simulated humanoid torsos (tavis-head-gr1t2,tavis-head-reachy2), and a stationary bimanual dexterous-hand rig (trex_dataset). observation.statedimensionality ranges from 23 (well,HIW-500-LeRobot's primary 29-DoF joint vector, with a separate 23-dim whole-body-control vector) up to 216 (AgiBotWorld2026, which also carries RL collection signals) — roughly a 9x spread across datasets that all fall under "humanoid/bimanual manipulation."tavis-head-gr1t2andtavis-head-reachy2run the same 5-task benchmark on two different humanoid torsos — the cleanest pair in this post for seeing embodiment heterogeneity in isolation, since the task and camera setup are held constant and only the robot's own DoF count changes (44-dim vs. 36-dim state).- 3 of 5 are real-world teleoperation (
AgiBotWorld2026,HIW-500-LeRobot,trex_dataset); 2 are simulation (tavis-head-gr1t2,tavis-head-reachy2, both built in NVIDIA IsaacLab). - Licensing: MIT (
trex_dataset), CC-BY-4.0 (HIW-500-LeRobot, bothtavis-head-*), CC-BY-NC-SA-4.0 (AgiBotWorld2026, non-commercial only).
What is embodiment heterogeneity?
Different robots have different action spaces (a 7-DoF single arm vs. a 14-DoF bimanual rig vs. a humanoid's whole-body state), different proprioceptive observation spaces (varying joint counts and sensor configurations), and different sensor modalities (RGB, depth, multiple viewpoints, tactile).
The same action vector induces different physical motion on different robots because end-effector coordinates translate differently depending on kinematic structure—and camera observations vary by mount position, resolution, and field of view on top of that. Coordinate frames aren't even guaranteed to agree: "forward" for one robot's base frame can be "left" for another's.
The field's general answer is coarse alignment.
For example, Open X-Embodiment normalizes everything to a unified 7-DoF end-effector representation (position x/y/z, orientation roll/pitch/yaw, gripper state), then de-normalizes it per embodiment at training time. That's a real compromise: the same action values might mean an absolute position for one robot and a relative velocity for another, and camera observations still don't line up. But that's what makes cross-embodiment learning possible at all, and models trained on that coarsely aligned diversity have been shown to outperform narrowly targeted single-embodiment counterparts, even when evaluated on a single embodiment.
None of the 5 datasets below have had that coarse alignment applied.
They're raw, as each source team designed them. That's deliberate for this post: seeing the unaligned heterogeneity first is what makes clear why Open X-Embodiment-style alignment work exists in the first place.
The embodiment spread
| Dataset | Robot | observation.state dim | action dim | Camera streams | Distinguishing sensing |
|---|---|---|---|---|---|
| HIW-500-LeRobot | Unitree G1 (real) | 29 (+ separate 23-dim observation.state.wbc) | 23 | 3 | Whole-body-control vector (pivot velocity/pose + dual end-effector pose) |
| tavis-head-reachy2 | Reachy2 (sim) | 36 | 19 | 4 | Active-vision head (pan/tilt neck) |
| tavis-head-gr1t2 | GR1T2 (sim) | 44 | 19 | 4 | Active-vision head (pan/tilt neck) |
| trex_dataset | Dexmate Vega-1 + Sharpa Wave hands (real) | 58 | 58 | 3 RGB + 20 tactile | Per-fingertip 6-axis tactile wrench |
| AgiBotWorld2026 | AgiBot G2 (real) | 216 | 44 | 7 | RL collection signals (reward/done/succeed/intervened) |
Reading this table left to right isn't just "more DoF, bigger number."
Each jump reflects a different design decision about what the platform needs to represent:
HIW-500-LeRobot's vector is almost entirely joint angles because it's a legged, whole-body-controlled humanoid where balance and locomotion state matter as much as arm position. trex_dataset's state and action vectors are identical in size and layout ([L_arm 7 | L_hand 22 | R_arm 7 | R_hand 22]) because every one of its 58 dimensions is a commanded joint position — dexterous hands alone eat 44 of those 58 dimensions. AgiBotWorld2026's 216-dim vector is the largest here not because the robot has more joints than the others, but because it's the only dataset in this post collected via reinforcement-learning-style rollout, and RL collection signals get concatenated in alongside camera extrinsics and full-body state.The
tavis-head-gr1t2/tavis-head-reachy2 pair is the cleanest illustration of pure embodiment heterogeneity in this table, because it's the only case here where the task, camera rig, and collection method are held constant and *only the robot changes*. Same 5 tasks (ClutterPickCubeTask, ClutterPickLiftTask, ConditionalPickTask, MultiShelfScanTask, WaitThenActTask), same 4-camera setup, same 60fps, same VR teleoperation pipeline in IsaacLab — and the state vector still differs by 8 dimensions (44 vs. 36) because GR1T2 and Reachy2 simply have different kinematic structures.If you want one policy that works on both, you'd need exactly the kind of per-embodiment de-normalization Open X-Embodiment does at the field level, in miniature, for these two robots alone.
The 5 humanoid LeRobot datasets at a glance
Keep it honest. "Episodes" reflects what's in each live FiftyOne export, not the full upstream source.
tavis-head-gr1t2 and tavis-head-reachy2 are complete (800/800 each); the other 3 are disk-budget-constrained subsets of much larger real-world sources.| Dataset | Description | License | Real or Sim |
|---|---|---|---|
| AgiBotWorld2026 | Real-world dual-arm humanoid manipulation collected via RL-style human-intervention rollout on the AgiBot G2 platform | CC-BY-NC-SA-4.0 | Real |
| HIW-500-LeRobot | Whole-body Unitree G1 humanoid teleoperation performing household chores across real homes in Southeast Asia | CC-BY-4.0 | Real |
| tavis-head-gr1t2 | Simulated active-vision humanoid benchmark (GR1T2) comparing head-mounted vs. fixed cameras on identical manipulation tasks | CC-BY-4.0 | Sim |
| tavis-head-reachy2 | The same active-vision benchmark as tavis-head-gr1t2, run on a Reachy2 humanoid instead | CC-BY-4.0 | Sim |
| trex_dataset | Bimanual dexterous manipulation with real per-fingertip tactile sensing on a Dexmate Vega-1 robot | MIT | Real |
The 5 humanoid LeRobot datasets
AgiBotWorld2026 — Real-world dual-arm manipulation with RL collection signals
- FiftyOne Format Dataset on Hugging Face: Voxel51/AgiBotWorld2026-sample-10ep
- Try it in a Hugging Face Space: harpreetsahota/AgiBotWorld2026-sample-10ep
- Original Dataset: agibot-world/AgiBotWorld2026 (102,078 episodes, 13.6TB)
- Additional Resources: Project homepage: agibot-world.com
- License: CC-BY-NC-SA-4.0
As one of the most expansive public initiatives in real-world humanoid manipulation, AgiBot World uses the dual-arm AgiBot G2 platform to collect long-horizon tasks in real-world offices, homes, and retail environments rather than controlled laboratory environments. The 2026 dataset release structures this data into three primary collection modalities: standard imitation learning, RL-based rollout featuring human intervention (HG-DAgger), and multi-signal interaction logs. This FiftyOne export focuses exclusively on the HG-DAgger reinforcement learning split, capturing a policy mid-training alongside reward metrics and human-correction markers rather than polished expert trajectories. All 10 exported episodes are derived from a single task archive (
ReinforcementLearning/Home/task_12192/HG-DAgger, "Insert the loose change into the coin slot of the piggy bank dagger").Data Breakdown & Features
- RL Collection Signals: Unique to the RL framework, each frame includes
reward,done,succeed,intervened(indicating active human correction),truncated, andaction_prob. - Camera Streams: 7 visual channels including
top_head,hand_left,hand_right, two head fisheyes, a rear fisheye, and ahead_depthstream (configured withdtype: videoandgray16bepixel format, allowing depth information to be ingested like standard RGB). - State & Action Vectors: A 216-dimensional
observation.statevector and a 44-dimensionalactionvector.
HIW-500-LeRobot — Humanoid teleoperation in real homes, sampled across all 11 tasks
- FiftyOne Format Dataset on Hugging Face: Voxel51/HIW-500-LeRobot-48ep
- Try it in a Hugging Face Space: harpreetsahota/HIW-500-LeRobot-48ep
- Original Dataset: BitRobot/HIW-500-LeRobot (23,743 episodes, ~2TB); raw ROS bag / MCAP recordings of this same data are separately published at
BitRobot/HIW-500 - Additional Resources: Project page
- License: CC-BY-4.0
The Humanoids In-the-Wild dataset delivers precisely what its title promises: hundreds of hours capturing a Unitree G1 humanoid carrying out everyday domestic chores across authentic Southeast Asian residences. It avoids controlled motion-capture setups and uniform apartment stages; instead, each episode introduces shifting floor plans, variable lighting conditions, physical clutter, and distinct operator teleoperation techniques. To ensure broad coverage rather than pulling an isolated single-task shard, we gathered 48 episodes across 6 distinct shard groups, capturing all 11 task categories—ranging from "setting the table" (15 episodes) to "hang hanger" (1 episode).
Available fields:
- Video Streams: 3 streams in AV1 format (480×1280
headcamera, 480×640left_wrist, and 480×640right_wristviews). - Observation State: A 29-DoF
observation.statecovering hips, knees, ankles, waist, shoulders, elbows, and wrists. - Whole-Body Control: A 23-dimensional
observation.state.wbcvector covering pivot pose/velocity, dual end-effector poses, and trigger/squeeze inputs. - Action Vector: A corresponding 23-dimensional
actionvector.
Unique Structure: The only dataset among the five highlighted where locomotion/stability controls (hip, knee, and ankle joint data) share a vector with manipulation attributes due to its full legged humanoid design.
tavis-head-gr1t2 and tavis-head-reachy2 — Same benchmark, two robots, different state size
- FiftyOne Format Dataset on Hugging Face: Voxel51/tavis-head-gr1t2-800ep · Voxel51/tavis-head-reachy2-800ep
- Try it in a Hugging Face Space: harpreetsahota/tavis-head-gr1t2-800ep · harpreetsahota/tavis-head-reachy2-800ep
- Original Dataset: tavis-benchmark org (both are complete, 800/800 episodes each)
- Additional Resources: Paper (arXiv:2605.07943) · Code · Paired π₀ checkpoints for head-camera vs. fixed-camera comparison, published under the same
tavis-benchmarkorg - License: CC-BY-4.0
The TAVIS benchmark isolates a critical question often overlooked by robot-learning datasets: does active head and camera movement directly improve manipulation performance, and can this advantage be quantified? To evaluate this, the benchmark records identical tasks using matching four-camera setups across two distinct humanoid torsos: Fourier's GR1T2 and Pollen's Reachy2. Every episode captures parallel streams from both a head-mounted active vision camera and a fixed baseline camera, enabling direct policy comparisons with head-camera integration enabled or disabled using the same demonstration data.
Both platforms evaluate five shared task classes (split into 250/250/100/100/100 episodes):
ClutterPickCubeTaskClutterPickLiftTaskConditionalPickTaskMultiShelfScanTaskWaitThenActTask
All demonstrations were gathered in NVIDIA IsaacLab using VR teleoperation recorded at 60fps across identical four-camera configurations, making this dataset the premier benchmark for studying embodiment heterogeneity under controlled conditions.
Dataset Specifications & Structure
- Visual Observations: Four 480×640 AV1 camera streams captured at 60fps (
OBS_HEAD,OBS_FIXED,OBS_WRIST_LEFT, andOBS_WRIST_RIGHT). - Action Space: A unified 19-dimensional action vector governing bimanual arm motion, neck movement, and gripper control across both robots.
- State Representation: Proprioceptive state (
observation.state) varies by kinematic structure—44 dimensions for GR1T2 versus 36 dimensions for Reachy2—meaning these vectors cannot be used interchangeably despite the shared action space. - Pose & Language Annotations: Includes dedicated left/right end-effector position and orientation attributes, alongside per-frame
language_instructionprompts (featuring 33 distinct instruction strings for Reachy2).
To support head-camera versus fixed-camera evaluation, the official
tavis-benchmark organization provides paired π₀ baseline checkpoints.trex_dataset — Bimanual dexterous manipulation with fingertip tactile sensing
- FiftyOne Format Dataset on Hugging Face: Voxel51/trex-dataset-23ep
- Try it in a Hugging Face Space: harpreetsahota/trex-dataset-23ep
- Original Dataset: zekaiwang/trex_dataset (5,464 episodes, 1.5TB)
- Additional Resources: Paper (arXiv:2606.17055) · Project page
- License: MIT
The T-Rex dataset highlights an exceptionally dense sensing modality rarely captured in robotic manipulation: authentic per-fingertip tactile feedback. Built using a stationary bimanual Dexmate Vega-1 setup equipped with Sharpa Wave dexterous hands, each hand features five dedicated tactile sensors. Every one of its 58 state values corresponds strictly to arm or hand joint positions, as the robot's base, torso, and head remained stationary during collection—distinguishing it from the mobile or legged systems featured elsewhere. Teleoperated with Manus gloves and VIVE trackers across 207 distinct objects and 22 motor primitives, this 23-episode slice represents the initial contiguous release of the larger dataset.
Key Data Fields:
- Visual Streams: 3 RGB feeds (
head_left,left_wrist,right_wristat 640×360). - Tactile Video Feeds: 20 distinct streams—providing a raw sensor image alongside an estimated deformation map for every fingertip (2 hands × 5 fingers × 2 stream types).
- Kinematic Vectors: 58-dimensional observation.state and matching action vectors structured as
[L_arm 7 | L_hand 22 | R_arm 7 | R_hand 22](both share identical dimensions because both record joint positions). - Tactile Force Measurements: A 60-dimensional
observation.tactile_forcevector capturing a 6-axis wrench per fingertip—a rich modality unique to this dataset among all five showcased.
Decoding Technicalities: To preserve exact physical measurements, the 20 tactile video channels are compressed losslessly via
libx264 -qp 0. This enforces the H.264 High 4:4:4 Predictive profile, which standard web browsers cannot natively render in an HTML <video> tag, so they lack video thumbnails in standard web interfaces. Standard decoding workflows in FiftyOne, ffmpeg, PyAV, or torchcodec handle these streams smoothly, while the 3 primary RGB channels use conventional yuv420p encoding and preview seamlessly everywhere.Frequently Asked Questions
It's the fact that different robots have different action spaces, proprioceptive observation spaces, and sensor modalities — a 7-DoF arm, a 14-DoF bimanual rig, and a legged humanoid's whole-body state simply don't share a representation. It matters because you can't concatenate datasets across embodiments without some form of alignment; a policy trained on one robot's 216-dim state vector has no native way to consume another robot's 36-dim vector, even if both robots are "doing the same task."
Not directly — none of them share a common action or state representation.
tavis-head-gr1t2 and tavis-head-reachy2 are the closest thing to compatible here (identical 19-dim action space, same task definitions), but even they differ in observation.state size (44 vs. 36). Combining any of these 5 for cross-embodiment training would require Open X-Embodiment-style coarse alignment — normalizing to a shared representation (e.g. a unified end-effector pose vector) and accepting that the same action values mean physically different things per robot.Because GR1T2 and Reachy2 are physically different robots with different degrees of freedom, even though the TAVIS benchmark holds the task, camera rig, fps, and collection method constant across both. The 44-vs-36 dimension gap in
observation.state is purely a function of each robot's kinematic structure—nothing about the task changed.tavis-head-gr1t2 and tavis-head-reachy2 share an identical 19-dim action space and the same 5 task definitions, so a policy that only consumes actions (not full proprioceptive state) could plausibly train across both with minimal remapping. Every other pair in this post differs in both state and action dimensionality, camera count, and often collection method.Real:
AgiBotWorld2026, HIW-500-LeRobot, trex_dataset. Simulated (NVIDIA IsaacLab): tavis-head-gr1t2, tavis-head-reachy2.MIT (
trex_dataset) is unrestricted. CC-BY-4.0 (HIW-500-LeRobot, tavis-head-gr1t2, tavis-head-reachy2) requires attribution but allows commercial use. CC-BY-NC-SA-4.0 (AgiBotWorld2026) is non-commercial only.Harpreet Sahota
Harpreet is Hacker-in-Residence at Voxel51, where he turns cutting-edge AI ideas into open-source prototypes and demos that push the boundaries of deep learning. From building tools that inspire to creating content that educates, Harpreet helps the AI community level up—one wild idea at a time.
See all articles by Harpreet Sahota
Talk to an AI expert
Loading related posts...
© 2026 Voxel51 All Rights Reserved
