LeRobotDataset
LeRobotDataset is the dataset format used by Hugging Face's LeRobot library to store robot demonstrations. It organizes data into episodes, storing joint states, actions, and timestamps in Parquet files, camera streams in MP4 video, and the schema and episode boundaries in metadata files. The current version, v3.0, packs many episodes into each file and can be streamed from the Hugging Face Hub.
In this article
Building visual or physical AI?
Let's talk.
What is LeRobotDataset?
LeRobotDataset is the standard way LeRobot stores robot learning data, and the format most community robot datasets on the Hugging Face Hub use. Each dataset is a folder of three kinds of files. Tabular data, such as the robot's joint positions and the actions it was commanded to take, lives in Apache Parquet files. Camera frames are encoded as MP4 video, with separate files for each camera. Metadata files describe every feature, the frame rate, the natural-language tasks, and normalization statistics.
Every frame carries a timestamp, a frame index within its episode, and an episode index, and features follow a naming convention: observation.state for proprioceptive state, action for commanded actions, and observation.images followed by a camera name for each video stream. In version 3.0, an episode is a slice of shared files rather than a file of its own, located by offsets in the episode metadata. That change lets a single dataset hold far more episodes without overwhelming the file system.
Key takeaways
- LeRobotDataset stores robot demonstrations as episodes of synchronized state, action, and video data.
- Tabular data lives in Parquet, camera streams in MP4, and the schema, tasks, and episode boundaries in metadata files.
- Version 3.0 packs many episodes into each file and supports streaming from the Hugging Face Hub. Version 2.x datasets need conversion before tools that read only v3.0 can load them.
What's inside a LeRobot dataset
| File or folder | What it holds |
|---|---|
| meta/info.json | The schema: each feature's data type and shape, the frame rate, and file path templates |
| meta/stats.json | Dataset-wide statistics used to normalize inputs during training |
| meta/tasks.parquet | Natural-language task descriptions, each mapped to a task index |
| meta/episodes/ | Per-episode length, tasks, statistics, and offsets into the shared files |
| data/ | Frame-by-frame tabular data in Parquet, many episodes per file |
| videos/ | Encoded camera streams in MP4, organized by camera, many episodes per file |
How it works
When LeRobot records a demonstration, it writes each frame's state, action, and timestamp to Parquet and encodes each camera stream to video. When the recording finishes, the metadata is updated with the new episode's length, task, and position in the shared files. At training time, LeRobot loads the dataset as a PyTorch dataset, decodes only the video frames it needs, and can return windows of past observations or future actions around any frame using time offsets in seconds.
Why it matters
LeRobotDataset matters because a shared format is what lets robot data be pooled. Datasets recorded on different robots, in different labs, can be loaded with the same code, combined into training mixes, and used to pretrain models such as SmolVLA. The format also makes data quality visible: inconsistent camera names, mismatched feature shapes, and noisy task descriptions show up directly in the metadata, which is where curation starts.
Frequently asked questions
What changed between LeRobotDataset v2.1 and v3.0?
Version 2.1 stored one Parquet file and one MP4 file per episode, with per-episode statistics in JSON Lines files. Version 3.0 packs many episodes into each file, moves episode metadata into Parquet, and adds streaming from the Hugging Face Hub. LeRobot includes a script to convert v2.1 datasets to v3.0.
Can a LeRobot dataset be streamed without downloading it?
Yes. Version 3.0 datasets can be streamed from the Hugging Face Hub, which avoids downloading datasets that can run to hundreds of gigabytes.
What tools can read LeRobotDataset?
The LeRobot library reads it natively, and Hugging Face hosts an online dataset visualizer. Other tools, including FiftyOne, load LeRobot v3.0 datasets for inspection and curation.
Related terms
Jesse Mostipak
Director of Growth
Jesse Mostipak is the Director of Growth at Voxel51, where the work is helping humans find and trust what the brand knows, and teaching the Google knowledge graph and the LLMs answering on their behalf to do the same. That question, how knowledge gets built inside a system, is one Jesse has been chasing for years. Earlier versions of it ran through a New York City high school science classroom, data science and machine learning, and developer relations at Kaggle, Posit (formerly RStudio), and Baseten. The answer doesn't change much depending on whether the learner is a teenager, a software engineer, or a knowledge graph. Jesse holds a Master's in Education from CUNY Hunter College. LinkedIn
See all articles by Jesse Mostipak
Last updated October 8, 2026
Building visual or physical AI?
Let's talk.
© 2026 Voxel51 All Rights Reserved