How to Import LeRobot Datasets into FiftyOne Using Agent Skills
Sep 16, 2026
• 6 min read
Author
Harpreet Sahota
Harpreet is Hacker-in-Residence at Voxel51, where he turns cutting-edge AI ideas into open-source prototypes and demos that push the boundaries of deep learning. From building tools that inspire to creating content that educates, Harpreet helps the AI community level up—one wild idea at a time.
See all articles by Harpreet Sahota
In this article
Talk to an AI expert
A standardized dataset format solved robot learning’s data fragmentation problem. Here’s how to get one of those datasets into FiftyOne without writing a parser, and what happens when the version doesn’t match what you expect.
Robot learning needs demonstration data the way language models need text, but until recently every lab and vendor recorded it differently: proprietary formats per robot, inconsistent schemas, no shared way to index or search across datasets. Hugging Face built
LeRobotDataset to fix that: one standardized format (Parquet for state and action, MP4 for camera feeds) across radically different embodiments, from SO-100 arms to bimanual ALOHA rigs to industrial Kuka arms.The standardization paid off measurably. SmolVLA, a 450M-parameter policy, was pretrained only on community-uploaded
lerobot-tagged datasets on the Hub, and that alone lifted SO-100 task success from 51.7% to 78.3% (source). The format itself had to evolve under that success: the original v2.x layout stored one Parquet file and one MP4 per episode, which started hitting filesystem limits once community datasets reached millions of episodes. lerobot v0.4.0 shipped v3.0 in response, packing many episodes into shared chunked files and supporting streaming without a full download first.FiftyOne ships
fo.types.LeRobotDataset as a first-class import path, built around one idea: a sample is an episode, not a file. The fiftyone-lerobot-import agent skill wraps that native support with the parts a script usually skips: detecting whether a Hub dataset is still on v2.x, running the documented conversion before import, and refusing to create a dataset or install a package without asking first.Key takeaways
- The fiftyone-lerobot-import agent skill imports LeRobotDataset v3.0 recordings into FiftyOne, where one sample is one episode rather than one file.
- The skill auto-detects LeRobot's older v2.x layout (
meta/episodes.jsonl,meta/tasks.jsonl) and runs the documented conversion to v3.0 before importing, instead of failing on an unsupported-version error. - Two explicit stop gates hold the run, one before installing any package (pyarrow, huggingface_hub, lerobot) and one before creating the FiftyOne dataset.
- Tested end to end against
lerobot/aloha_sim_insertion_human, a real v2.1-tagged Hub dataset, which converted to v3.0 and imported as 50 episodes of a bimanual ALOHA arm with zero episodes skipped. - The skill is a thin layer over the open-source FiftyOne SDK (
fo.Dataset.from_dir,dataset.add_dir), so the same import runs identically from a chat prompt or a checked-in script.
Part one: what is an agent skill, actually?
Strip away the buzzwords and a skill is a folder. Inside it is a markdown file that tells an AI coding agent, step by step, how an expert would do a specific job: which files to check first, which command to run, which traps to avoid, and how to tell success from failure. The agent reads that folder like a new hire reading a training guide, then does the work itself in your terminal, on your data.
Voxel51 packages this kind of runbook for FiftyOne into an open-source repository, installed with one line:
curl -sL skil.sh | sh -s -- voxel51/fiftyone-skillsThe installer clones the repo, discovers every skill inside it, and lets you pick which ones your agent should have access to going forward.
This post is about
fiftyone-lerobot-import. Point it at a local LeRobot directory or a Hugging Face Hub repo id, and it detects the version, converts if needed, and builds the FiftyOne dataset. That’s a different problem from the one fiftyone-dataset-import’s MCAP path solves. MCAP’s question is a schema scan: what kind of sensor data is this? LeRobot’s question is about versioning: what version of this format is this, and does it need conversion first?Part Two: A Real LeRobot Dataset, End to End
A skill only proves itself on data that actually exercises its hardest path. A dataset already sitting on v3.0 would import cleanly and tell you nothing about the conversion logic, so we looked for a dataset still tagged v2.1 on the Hub.
The dataset: lerobot/aloha_sim_insertion_human
lerobot/aloha_sim_insertion_human is one of LeRobot’s official example repos: a bimanual ALOHA arm running a simulated peg-insertion task, one task label (“Insert the peg into the socket.”), 50 episodes, 50 fps, 25,000 total frames.| Detail | Value |
|---|---|
| Repo | lerobot/aloha_sim_insertion_human |
| Revision pulled | v2.1 |
| Episodes | 50 |
| FPS | 50 |
| robot_type | aloha |
| Camera | observation.images.top (video, av1 codec, 480×640) |
| observation.state / action dims | 14 / 14 (bimanual: waist, shoulder, elbow, forearm roll, wrist angle, wrist rotate, gripper, same layout on both arms) |
| Task | “Insert the peg into the socket.” |
We pulled the Hub’s
v2.1 tag specifically, not main (already v3.0), so the conversion path would get exercised for real instead of simulated.The prompt and the version-detection moment
The prompt to the agent was one sentence: import
lerobot/aloha_sim_insertion_human into FiftyOne.The skill’s inspection step reads
meta/info.json and finds codebase_version: v2.1, along with layout markers like meta/episodes.jsonl and meta/tasks.jsonl. Instead of failing on unsupported versions, the skill detects the older format, hits the packages stop gate, and runs the conversion script (python-m lerobot. scripts. convert_dataset_v21_to_v30) to update the dataset to v3.0, ensuring reliable import even with legacy data.- Hits the packages stop gate and asks before installing
lerobot(needed only for the conversion). - Runs the conversion:
python -m lerobot.scripts.convert_dataset_v21_to_v30 --repo-id lerobot/aloha_sim_insertion_human --root <path> --push-to-hub=false. - Keep it honest. The installed lerobot release (v0.6.2) didn’t expose the converter at the documented
lerobot.datasets.v30.*module path; it was still at the olderlerobot.scripts.*path. The skill’s own troubleshooting notes already anticipate this exact fallback, and it’s what happened on the first real run, not a hypothetical edge case. - Re-checks
meta/info.json, confirmscodebase_version: v3.0, and the new layout:meta/episodes/chunk-000,meta/tasks.parquet,meta/stats.json. - Hits the creation stop gate and presents the full import plan before touching anything:
LeRobot Import Plan for lerobot/aloha_sim_insertion_human:
Source:
- codebase_version: v3.0 (converted from v2.1: yes)
- 50 episodes, 50 fps, robot_type: aloha
- camera: observation.images.top (video, av1)
- observation.state: 14 dims, action: 14 dims
- tasks: 1 ("Insert the peg into the socket.")
Import:
- Dataset name: lerobot-aloha-sim-insertion-human
- Episodes: all
- Persistent: yes
Proceed with import? (yes/no)The payoff: 50 episodes, zero skipped
dataset.media_type comes back "multimodal". All 50 episodes import; dataset.info["lerobot"]["skipped_episodes"] is an empty list: a clean import, worth stating plainly rather than manufacturing a gap that isn’t there. Every episode runs exactly 10 seconds (dataset.bounds("duration") returns (10.0, 10.0)), which checks out at 50 fps × 500 frames/episode.Opening episode 0 in the App shows the
observation.images.top tile: two robot arms working over a blue and red object, with the episode header reading “Insert the peg into the socket. · 10s.” That’s what “one sample equals one episode” looks like in the viewer, not an abstraction: a single click surfaces the whole 10-second demonstration, not one video frame.Part Three: The Same LeRobot Import, in Code
A chat prompt is a good way to explore a dataset you haven’t seen before, but a pipeline that runs on every new upload needs code, not a conversation. Everything the skill did above sits on top of the open-source FiftyOne SDK. Here’s the full pipeline, runnable from a fresh environment against the same Hub dataset used in Part Two.
Step 1: Install the dependencies
pip install "pyarrow>=10.0.0" huggingface_hub lerobotlerobot is only needed for the v2.1 → v3.0 conversion below. A dataset already on codebase_version: v3.0 only needs pyarrow and huggingface_hub.Step 2: Download from the Hub
fo.Dataset.from_dir reads from a local directory, not directly from the Hub, so pull the files down first:
from huggingface_hub import snapshot_download
local_dir = snapshot_download(
repo_id="lerobot/aloha_sim_insertion_human",
repo_type="dataset",
revision="v2.1",
local_dir="/abs/path/lerobot/aloha_sim_insertion_human",
)Step 3: Convert v2.1 to v3.0
python -m lerobot.scripts.convert_dataset_v21_to_v30 \
--repo-id lerobot/aloha_sim_insertion_human \
--root /abs/path/lerobot/aloha_sim_insertion_human \
--push-to-hub=falseSkip this step for any dataset already tagged
v3.0 on the Hub; for example, pull revision="v3.0" in Step 2 instead.Step 4: Import into FiftyOne
import fiftyone as fo
dataset = fo.Dataset.from_dir(
dataset_dir=local_dir,
dataset_type=fo.types.LeRobotDataset,
name="lerobot-aloha-sim-insertion-human",
persistent=True,
)
assert dataset.media_type == "multimodal"
fo.launch_app(dataset)Adding a second recording session to the same dataset is the same four steps repeated against the new source directory, then
dataset.add_dir(dataset_dir=..., dataset_type=fo.types.LeRobotDataset) instead of from_dir.No Parquet reader, no video decode loop, no custom viewer. The importer reads
meta/info.json and meta/episodes/*.parquet, nothing in the video files, so it stays fast even on large sources. The skill’s stop gates are a conversation; the SDK calls underneath are the same ones either way.Where This Leaves You
Every team re-solves “is my data in the format my tools expect” by hand, one script at a time, and LeRobot is just the latest format to need that check.
fiftyone-lerobot-import turns that check into something an agent already knows how to run correctly against your files, every time.- Dataset:
lerobot/aloha_sim_insertion_human, part of LeRobot’s official example datasets, Hugging Face - Skill repo:
voxel51/fiftyone-skills, installed the same one-line way as Part One
Frequently Asked Questions
Two ways: ask an agent with the
fiftyone-lerobot-import skill, or call fo.Dataset.from_dir(..., dataset_type=fo.types.LeRobotDataset) directly. Both require the dataset to be on codebase_version: v3.0.v2.x stores one Parquet file and one MP4 per episode. v3.0 packs many episodes into shared chunked files and supports streaming.
fo.types.LeRobotDataset only supports v3.0. v2.x sources need conversion first.Only for converting a v2.x source to v3.0. A straight v3.0 import needs only
pyarrow>=10.0.0.One sample per episode, not per file, with
media_type == "multimodal" and fields like episode_index, task, duration, and fps.They’re reported under
dataset.info["lerobot"]["skipped_episodes"] instead of being silently dropped or silently accepted.Harpreet Sahota
Harpreet is Hacker-in-Residence at Voxel51, where he turns cutting-edge AI ideas into open-source prototypes and demos that push the boundaries of deep learning. From building tools that inspire to creating content that educates, Harpreet helps the AI community level up—one wild idea at a time.
See all articles by Harpreet Sahota
Talk to an AI expert
Loading related posts...
© 2026 Voxel51 All Rights Reserved
