LeRobot is Hugging Face's open source library for real-world robot learning. It combines a standard dataset format, a catalog of trainable policies, and a common interface to robot hardware. This guide covers how the LeRobot workflow fits together, from recording demonstrations and structuring LeRobot datasets to curating them, training policies, and evaluating the results.
What is LeRobot?
LeRobot is an open source library from Hugging Face for training robots to act in the real world. It gives robot learning teams three things they used to build themselves: the LeRobotDataset format for storing demonstrations, policies from imitation learning models to vision-language-action (VLA) models, and one interface for recording robot data and deploying trained policies.
The library is written in PyTorch, released under the Apache 2.0 license, and maintained by Hugging Face's LeRobot team with a large open source community of contributors.
LeRobot exists because robot learning has been fragmented. Each lab recorded demonstrations in its own format, trained with its own code, and ran on hardware few others owned, which made results hard to reproduce and datasets hard to combine. LeRobot lowers that barrier with low-cost open hardware such as the SO-101 arm, a shared dataset format, and the Hugging Face Hub as a common place to publish data and trained policies. As of September 2025, the LeRobot paper counted more than 16,000 community datasets from over 2,200 contributors on the Hub. By November 2026, the Hub listed more than 80,000 datasets tagged LeRobot.
Hugging Face's LeRobot documentation is the authoritative reference for installation, hardware assembly, and training commands. This guide maps how the pieces of the LeRobot workflow connect, explains the data concepts underneath them, and links to the official documentation and to hands-on LeRobot tutorials from the Voxel51 blog at each step.
LeRobot organizes robot learning as a workflow. A person teleoperates a robot, or a simulator generates demonstrations, and each attempt at a task is recorded as an episode. Episodes are written into a LeRobot dataset and shared on the Hugging Face Hub. Engineers inspect the episodes, curate the ones worth training on, and annotate them with task descriptions. Policies trained on the curated dataset are evaluated in simulation benchmarks and on real robots, and the failures found in evaluation become the next targets for recording and curation. The diagram below summarizes the workflow, and the rest of this guide explores each stage.
LeRobot is Hugging Face's open source library for real-world robot learning, combining a dataset format, trainable policies, and a common hardware interface.
A LeRobot dataset stores demonstrations as episodes, with state and action data in Parquet files, camera streams in MP4 video, and metadata describing features, tasks, and episode boundaries.
The LeRobot workflow runs from recording through sharing, inspection, curation, annotation, training, and evaluation.
In the LeRobot workflow, evaluation failures feed back into recording and curation, which makes dataset quality part of the policy development loop.
LeRobot tutorials and guides from the Voxel51 blog
These posts cover LeRobot data from collection through curation and evaluation. Each section below links to more.
LeRobot support in Voxel51: native LeRobot support with synchronized playback, search, and temporal tags, and how LeRobot compares with MCAP.
LeRobot draws its vocabulary from robotics, reinforcement learning, and machine learning. These terms anchor the rest of this guide.
LeRobot: Hugging Face's open source library and dataset format for robot learning.
Episode: one complete attempt at a task, and the unit of data a LeRobot dataset stores. Also called a trajectory, rollout, or demonstration.
Teleoperation: controlling a robot remotely, which is how most LeRobot demonstrations are recorded.
Proprioception: the robot's sense of its own joint positions and forces, stored in LeRobot datasets as the observation.state feature.
Policy: the learned model, trained on a LeRobot dataset, that maps observations to actions and controls the robot.
Imitation learning: training a policy to reproduce recorded demonstrations, the most common use of LeRobot datasets.
Action chunking: predicting a short sequence of future actions at once instead of one action per step, as policies such as Action Chunking with Transformers (ACT) do.
Vision-language-action model (VLA): a model that maps camera input and a language instruction directly to robot actions, often pretrained on pooled LeRobot community datasets.
Embodiment: the physical form of a robot, including its joints, sensors, and action space.
Cross-embodiment: training one policy on LeRobot datasets from several robot types.
Policy evaluation: measuring how often a trained policy completes its task, usually reported as a success rate.
LeRobotDataset: the LeRobot dataset format, which stores episodes as Parquet state and action data, MP4 camera video, and metadata.
SO-101 robot arm: the low-cost, open source arm used to record most community LeRobot datasets, built as a leader and follower pair.
Leader-follower teleoperation: recording demonstrations by moving a leader arm that a follower arm mirrors joint for joint.
Task annotation: the natural-language task descriptions attached to episodes, which language-conditioned policies learn to follow.
Action Chunking with Transformers (ACT): an imitation learning policy that predicts short sequences of future actions, often the first policy trained on a new LeRobot dataset.
Diffusion Policy: an imitation learning policy that generates action sequences by denoising, which handles demonstrations with more than one valid way to act.
SmolVLA: Hugging Face's compact vision-language-action model, pretrained on community LeRobot datasets.
Human-in-the-loop data collection: recording a person's corrections while a policy runs, so new data targets the situations where the policy fails.
How is LeRobot data recorded? Robots and teleoperation
LeRobot data starts with a demonstration. In the most common setup, a person moves a leader arm by hand while a follower arm of the same design mirrors the motion, and LeRobot records the follower's joint positions, the commanded actions, and synchronized camera frames at a fixed frame rate. Each recorded attempt becomes one episode. The same recording pipeline works with other teleoperation devices, including gamepads, keyboards, phones, and VR headsets, and with trained policies running on their own, which lets teams record evaluation rollouts in the same format as their training data. Handheld recorders such as Grabette, from Hugging Face's Pollen Robotics, capture demonstrations without a robot at all and convert them into LeRobot datasets.
LeRobot supports a growing set of robots natively, from low-cost arms such as the SO-100, SO-101, and Koch to the LeKiwi mobile manipulator, the Reachy 2 and Unitree G1 humanoids, and the OpenArm bimanual platform. Third-party integrations extend the same interface to industrial arms such as the UFACTORY xArm and UR5e. The low-cost arms shape the dataset ecosystem most: the LeRobot paper puts an SO-101 arm at about €225, and SO-100 and SO-101 arms account for more than half of the community datasets on the Hub.
Simulation is the other major source of LeRobot data. LeRobot integrates simulated environments and benchmarks such as ALOHA, PushT, LIBERO, and Meta-World, so teams can generate demonstrations at scale and evaluate policies without a physical robot.
Hugging Face's recording guidance is a useful baseline for a first dataset: record at least 50 episodes of a simple task, keep the cameras fixed, and check that a person could complete the task by looking only at the camera images. Teams that prefer a graphical interface to the command line can use LeLab, LeRobot's GUI.
How is a LeRobot dataset structured? The LeRobotDataset format
A LeRobot dataset stores robot demonstrations in a layout built to be compact on disk and fast to stream during training. The current version of the format, LeRobotDataset v3.0, has three parts: tabular data, video, and metadata. Tabular data such as joint states, actions, and timestamps is stored in Apache Parquet files. Camera streams are encoded as MP4 video, with a separate set of files for each camera. Metadata files describe the dataset's schema, frame rate, tasks, episode boundaries, and normalization statistics.
Path
What it holds
meta/info.json
The schema: each feature's data type and shape, the frame rate, the codebase version, and path templates for data and video files
meta/stats.json
Dataset-wide statistics (mean, standard deviation, minimum, and maximum) used to normalize inputs during training
meta/tasks.parquet
The natural-language task descriptions, each mapped to an integer task index
meta/episodes/
Per-episode metadata: length, tasks, per-episode statistics, and offsets into the shared data and video files
data/
Frame-by-frame tabular data, with many episodes in each Parquet file
videos/<camera>/
Encoded camera streams, with many episodes in each MP4 file
The LeRobotDataset v3.0 directory layout, with what each metadata file and folder holds.
Every frame in a LeRobot dataset carries a timestamp within its episode, a frame index that restarts at zero for each episode, a global index across the dataset, an episode index, and a task index. Features follow a naming convention: observation.state holds the robot's proprioceptive state, action holds the commanded action, and observation.images followed by a camera name holds each camera stream. In v3.0, an episode is a slice of shared files rather than a file of its own, and the episode metadata records where each episode starts and ends in both the Parquet data and the video. LeRobot datasets load as PyTorch datasets and can be streamed directly from the Hugging Face Hub without downloading them first, which matters for datasets that run to hundreds of gigabytes.
LeRobot v2.1 vs. v3.0
The LeRobotDataset format has changed in ways that affect anyone working with older datasets. Version 2.0 stored one Parquet file and one MP4 file per episode. Version 2.1 kept that layout and added per-episode statistics. Version 3.0, announced in September 2025, packs many episodes into each file, moves episode metadata from JSON Lines files into Parquet, and adds streaming from the Hugging Face Hub.
LeRobotDataset v2.1
LeRobotDataset v3.0
File layout
One Parquet file and one MP4 file per episode
Many episodes per Parquet file and per MP4 file
Episode metadata
JSON Lines files
Parquet files with per-episode offsets
Statistics
Per-episode statistics in episodes_stats.jsonl
Global statistics plus per-episode statistics in the episode metadata
File count
Grows with every episode
Grows with data volume, not episode count
Hub access
Download, then load
Stream directly from the Hub
LeRobotDataset v2.1 and v3.0 compared by file layout, episode metadata, statistics, file count, and Hub access.
LeRobot includes a script that converts v2.1 datasets to v3.0, and current LeRobot versions won't load a v2.1 dataset until it is converted. As with any migration, load the converted dataset and spot-check several episodes, including their video, timestamps, and task descriptions, before training on it.
The Hugging Face Hub is where LeRobot datasets are published, discovered, and reused. Datasets recorded with LeRobot are tagged LeRobot on the Hub automatically, and an online visualizer lets anyone browse a published dataset's episodes in the browser before downloading anything. Publishing to a shared Hub turned individual teleoperation sessions into a community resource: small datasets recorded on inexpensive arms in labs, classrooms, and hackathons can be pooled into training mixes no single team could collect.
Pooled community datasets now train real models. SmolVLA, a 450-million-parameter VLA that Hugging Face released in 2025, was pretrained on 481 community datasets totaling about 22,900 episodes and 10.6 million frames. The Hub also hosts LeRobot-format ports of established research datasets, including the Open X-Embodiment collection and DROID, so teams can combine community data with large-scale lab data in one format.
Community data also brings community variance. Datasets recorded by different people on different setups name their cameras differently, describe tasks inconsistently, and vary in quality from one episode to the next. That variance is why inspection and curation sit between the Hub and training in the LeRobot workflow.
Inspecting a LeRobot dataset before training on it catches problems that a training loss curve never shows. A dataset can load without errors and still contain episodes with a dropped camera stream, a task description that reads "test," joint data recorded under an old calibration, or a camera labeled as the wrist view that shows the table from above. Watching episodes with every camera synchronized to the state and action data on one timeline is the fastest way to find these problems.
LeRobot ships its own visualization tools: a command-line viewer for local datasets and an online dataset visualizer on the Hugging Face Hub, which plays camera streams next to state and action plots. Questions that span a whole dataset, or several datasets at once, call for indexing the episodes in a dataset tool where teams can filter by robot type, task, and episode length, compare episodes side by side, and tag the moments that need attention.
Two checks pay off on almost every LeRobot dataset. Comparing the commanded action against the measured state shows how closely the robot tracked its instructions. Comparing each feature's values against the statistics in meta/stats.json surfaces out-of-range readings that point to a calibration or unit problem.
Curating LeRobot datasets: finding the episodes worth training on
Curation decides which episodes a policy learns from. Imitation learning reproduces whatever the demonstrations show, including hesitations, failed grasps, and idle pauses, so the quality and consistency of the episodes matter as much as their number. Curating a LeRobot dataset means finding the episodes to remove, fix, or rebalance, and it becomes essential once datasets from many contributors are combined.
Problem
What it looks like
How teams catch it
Noisy task descriptions
Empty, placeholder, or one-word task strings such as "test" or "desc"
Read the task list in meta/tasks.parquet, relabel with a vision-language model (VLM), and spot-check the results
Inconsistent camera names
The same key, such as images.laptop, means a top view in one dataset and a wrist view in another
Watch a sample episode from each source and map camera keys to one naming scheme before mixing datasets
Idle segments and failed attempts
The arm pauses, retries, or drops the object partway through an episode
Motion and smoothness scores, episode length outliers, and watching the flagged episodes
Very short or truncated episodes
Episodes with only a few frames, often from an interrupted recording
Filter on episode length in the episode metadata
Mismatched state and action dimensions
Datasets from the same robot type expose different numbers of joints
Compare feature shapes in meta/info.json across datasets
Timestamp and frame rate errors
Frame timestamps drift from the declared frame rate, which can raise timestamp errors when the dataset loads, or a recording loop runs slower than its target and the dataset still declares the target rate, because timestamps come from frame indices
Validate timestamps against the declared frame rate before training
Calibration changes
Episodes recorded before a calibration change replay incorrectly
Compare state ranges against meta/stats.json and group episodes by recording session
Common LeRobot dataset problems, what each looks like, and how teams catch it before training.
Curation tools for LeRobot datasets combine metadata filters, quality scores computed from the state and action streams, and embeddings computed from the camera streams, which group similar episodes and expose near-duplicates and gaps. Quality scores work best as triage that tells you which episodes to watch first, because the smoothest episode in a batch can still be a failure. The curation metrics guide explains what each LeRobot quality score measures.
In a 497-episode sample of the LeRobot Community Dataset, 495 episodes had a nearest neighbor from the same source dataset. Each contributor's setup shapes the data more than the task does.
Annotating LeRobot data: tasks, language, and subtasks
Annotation in a LeRobot dataset is mostly language and time. Every episode carries at least one task description, and that string is the instruction a language-conditioned policy or VLA learns to follow, so a vague or wrong task description teaches the wrong mapping from instruction to behavior. Beyond the episode-level task, teams annotate subtasks and events with start and end times, such as reach, grasp, and place, which supports training on long-horizon tasks and makes failures easier to locate.
In 2026, LeRobot added optional language columns to the dataset format, along with an annotation pipeline that uses a VLM to propose language annotations and a browser-based annotation tool on the Hub. Temporal tags applied during inspection, which attach a label to a time range inside an episode, serve a similar purpose during curation. They turn "something goes wrong around the four-second mark" into a record a team can filter on.
A temporal tag marks the moment the gripper closes, so the event can be searched and filtered later.
LeRobot includes implementations of many published policies, and all of them train from the same dataset format with the same training command. The policies fall into a few families.
Map camera images and a task instruction to actions, usually pretrained on many datasets and fine-tuned on a target robot
Reinforcement learning
HIL-SERL, TD-MPC
Improve a policy through trial and error, with rewards and optional human interventions
Reward models
SARM, ROBOMETER, TOPReward
Score progress toward a task, for evaluation and data filtering
LeRobot policy families, with examples of each and what they learn from LeRobot data.
Training reads more from a LeRobot dataset than the frames themselves. Normalization uses the statistics in meta/stats.json, so a corrupted statistic skews every input. Policies that use action chunking, such as ACT, sample sequences of future actions using time offsets relative to the current frame. VLAs pretrained with one set of camera names need each new dataset's camera keys mapped onto that set, which LeRobot handles with a rename map. Each of these depends on correct metadata, which is another reason inspection comes before training.
Mixing datasets across embodiments is where training and curation meet. A VLA trained on data from many robots generalizes only as well as its training mix allows, and balancing that mix across robot types, tasks, and environments is a curation decision.
A LeRobot policy is evaluated by running it. In simulation, LeRobot's evaluation tooling runs a policy across benchmark suites such as LIBERO, Meta-World, RoboTwin 2.0, and RoboCasa365 and reports success rates. On a physical robot, a rollout runs the policy on the real task, and LeRobot can record those rollouts as episodes in the same dataset format used for training. Human-in-the-loop modes let an operator take over when the policy goes wrong, and the corrections are recorded too.
A success rate alone does not show why a policy fails. Recording evaluation rollouts as LeRobot episodes makes failures inspectable with the same tools used on training data. A failed grasp can be compared against the training episodes that showed the same grasp, and a pattern of failures can point to a gap in the training data, such as an object position or lighting condition the demonstrations never covered. Human corrections recorded during rollouts become new training episodes aimed at exactly the situations where the policy struggled.
Saving confirmed failures as a view turns them into a targeted set for the next round of data collection.
Evaluation closes the LeRobot workflow. The scenarios that expose policy failures become the next demonstrations teams record, curate, and add to future training datasets.
The LeRobot workflow is the same for every robot it supports. What changes by embodiment is the shape of the data: how many joints the state vector holds, how many cameras record each episode, and how expensive each demonstration is to collect.
LeRobot for low-cost arms
Single-arm setups such as the SO-101 record a six-dimensional joint state and one or two camera streams, typically at 30 frames per second, and they produce most of the community data on the Hub. Their low cost makes them the usual entry point for imitation learning, and their volume makes consistency across contributors the main curation challenge. Explore: curation metrics for LeRobot datasets, SO-101 setup (Hugging Face docs).
LeRobot for humanoids and dexterous hands
Humanoid and dexterous-hand datasets carry much larger state and action vectors and more cameras. A simulated 60-frames-per-second humanoid episode can carry a 44-dimensional state vector and four video streams per frame, and two robots doing nearly identical tasks can record their own state as a 29-dimensional vector or a 216-dimensional one. Cross-embodiment training on these datasets is a schema alignment problem as much as a learning problem. Explore: 5 humanoid LeRobot datasets for VLA training, Unitree G1 (Hugging Face docs), humanoid robot.
LeRobot sits inside a broader ecosystem of robotics middleware, recording formats, dataset formats, and benchmarks. Many robots run on Robot Operating System 2 (ROS 2), which records sensor data to MCAP files: continuous, lossless logs of every message a robot's sensors produced. LeRobot datasets hold a different layer of data, synchronized episodes of camera, state, and action structured for training policies. Teams often record to MCAP, curate the recordings, and export the episodes worth training on into LeRobot format. Before LeRobot, robot learning datasets were most often distributed in RLDS, the episode-and-step format built on TensorFlow Datasets that Open X-Embodiment uses to pool more than a million trajectories from 22 robot embodiments. Other formats persist in specific communities, including HDF5 for ALOHA and robomimic datasets and Zarr for Diffusion Policy. Community converters translate between these formats and LeRobot. Robot descriptions use URDF, benchmarks such as LIBERO and Meta-World define how policies are compared, and the Hugging Face Hub is where LeRobot datasets and policies are shared.
Format
Layer
Structured around
Storage
Typical use
LeRobotDataset
Training dataset
Episodes of frames with state, action, and video
Parquet, MP4, and metadata files
Imitation learning and VLA training, shared on the Hugging Face Hub
RLDS
Training dataset
Episodes as ordered steps
TensorFlow Datasets records
Open X-Embodiment and the research models trained on it
HDF5
Training dataset
Arrays stored per episode file
HDF5
ALOHA and robomimic datasets
MCAP
Recording container
Timestamped message streams
Self-describing container with embedded schemas
ROS 2 recording, replay, and log analysis
LeRobotDataset, RLDS, HDF5, and MCAP compared by layer, structure, storage, and typical use.
You can run the inspection and curation steps in this guide without writing code. In the free tier of Voxel51, you can:
Browse the 20+ demo datasets and add LeRobot Community, 2,378 tabletop manipulation episodes from 117 community datasets. Each dataset you add uses one of three free dataset slots.
Play each episode's cameras on one timeline with its state and action values.
Filter, sort, and tag the episodes you don't trust, and mark bad moments with temporal tags.
Find more episodes that look like the failures you spotted, then save the rest as a clean subset.
Sign up with a work or .edu email, then start with the LeRobot Community demo. The LeRobot support post shows the same playback, search, and tagging on LeRobot data, and Try Voxel51 free covers what the free tier includes.
LeRobot is neither a robot nor a company. It is an open source software project maintained by Hugging Face. The name refers to the library, its dataset format, and the datasets and models shared on the Hugging Face Hub under the LeRobot tag. LeRobot runs on many robots, including the low-cost SO-101 arm, an open hardware design that the LeRobot documentation calls its flagship robot.
LeRobot includes a conversion script, convert_dataset_v21_to_v30, that rewrites a v2.1 dataset in the v3.0 layout. After converting, load the dataset and spot-check several episodes, including their video, timestamps, and task descriptions, before training on it. Current LeRobot versions won't load a v2.1 dataset until it is converted.
To check the quality of a LeRobot dataset, start with the metadata: confirm the frame rate, feature shapes, and camera names in meta/info.json, and read the task descriptions in meta/tasks.parquet. Then watch a sample of episodes from each source, compare commanded actions against measured state, and filter for very short episodes, idle segments, and failed attempts. Quality scores and embeddings help rank which episodes to review first when a dataset is too large to watch in full. The LeRobot curation metrics guide explains what each score measures.
LeRobot and RLDS are both episode-based formats for robot learning data. RLDS stores episodes as ordered steps in TensorFlow Datasets and is the format behind Open X-Embodiment. LeRobotDataset stores tabular data in Parquet and camera streams as compressed MP4 video, loads natively in PyTorch, and streams from the Hugging Face Hub. Community converters work in both directions, and Hugging Face hosts LeRobot-format ports of Open X-Embodiment datasets.
Yes, MCAP files can be converted to LeRobot format. MCAP files are raw recordings of every message a robot logged, while LeRobot datasets are episodes structured for training, so conversion means choosing which topics become observations and actions, cutting the recording into episodes, and resampling streams to a common frame rate. Teams typically curate MCAP recordings first and export only the episodes worth training on.
A timestamp error such as FrameTimestampError means LeRobot could not match the timestamps it expected from the tabular data to frames decoded from the video, within a small tolerance. Teams typically look for dropped or duplicated frames, frame rates that differ between streams, and rounding drift in recorded timestamps. Checking timestamps against the declared frame rate before training, and trimming or re-recording the affected episodes, resolves most cases.