Five Problems Hiding in 383 Robot Episodes: A LeRobot Dataset Quality Check

•
10 min read
Author
Headshot of Harpreet Sahota
Hacker-in-Residence
Harpreet is Hacker-in-Residence at Voxel51, where he turns cutting-edge AI ideas into open-source prototypes and demos that push the boundaries of deep learning. From building tools that inspire to creating content that educates, Harpreet helps the AI community level up—one wild idea at a time.
See all articles by Harpreet Sahota

Try Voxel51 for yourself

Open a LeRobot dataset in Voxel51 and find aborted takes, repeated layouts, hidden cube colors, and bystanders before you spend a GPU hour.
Episode 20 of svla_so100_stacking is one frame long. Open it, and you get a black screen, and that's all there is.
That episode is in an official LeRobot dataset on the Hugging Face Hub. It's listed alongside the 15-second takes where the arm stacks a red cube on a blue one, with nothing to mark it as broken.
Voxel51 grid of svla_so100_stacking episodes, with episode 20 opening to empty camera panes and a No data at this time placeholder.
Episode 20 of svla_so100_stacking holds one frame and no usable video.
Robot learning data is recorded by people, or by policies a person is supervising. People stop and take early and forget to delete it. They put the cube back in the same spot fifty times. They leave the camera running while someone walks behind the table. None of it announces itself. You find out when the policy fails, and by then you're debugging the wrong thing.
This post walks through one merged dataset, 383 episodes from seven sources, and the five problems we found in it by looking at the data.

Key takeaways

  • A LeRobot timestep is two camera images plus twelve numbers (six joint positions, six commanded positions), recorded 30 times a second. The dataset in this post has 143,799 of them across 383 episodes.
  • LeRobot v3 stores an episode as a row of offsets into shared files. Episode 30 of the stacking set is rows 13,062 to 13,532 of one Parquet file and seconds 435 to 451 of one MP4, so you can't browse the data by opening files.
  • Voxel51 loads each LeRobot episode as one sample, with fields you can filter and sort on (duration, task, robot_type, episode_index) and a reference back to the original video and Parquet files. Nothing is copied.
  • Five problems turned up in the LeRobot dataset: bad metadata (six aborted takes, ten task strings), repeated cube layouts in the sorting set, five cube colors hidden behind one task string, bystanders and a hand in frame, and a search that can't see what's missing. Each was found with a different tool: a filter, a distance, the embeddings plot, similarity search, and plain-language search.
  • None of the five LeRobot dataset checks needs a trained policy.

What does a robot record?

Episode 5 of svla_so100_pickplace playing in Voxel51, with the top camera and wrist camera side by side as the arm drops an orange cube into a box.
One LeRobot timestep pairs a top camera frame with a wrist camera frame, recorded 30 times a second.
Take one moment from one recording. One timestep.
There are two images, one from each camera. In the stacking set, one camera (top) looks down at the table, and one (wrist) rides on the gripper.
Then there are twelve numbers. Six are the state: where each joint is right now, from the shoulder to the gripper. LeRobot stores them as the observation.state feature. The other six are the action, stored as action: where each joint was told to go. State is where the arm is, action is where it was told to be, and the gap between them is the arm catching up.
Good to know. What is proprioception? Proprioception is a robot's internal sense of its own body: joint positions, velocities, torques, end-effector pose, and gripper state. A camera senses the outside world. Proprioception senses the machine itself, and a policy reads both at every step. In a LeRobot dataset, it's the observation.state feature.
Episode 1 of svla_so100_pickplace in Voxel51, with action and observation.state values and a plot where commanded and measured joint positions track each other.
State and action for each SO-100 joint on one timeline. The gap between the lines is the arm catching up.
An episode is one complete attempt at a task, and it's the unit a LeRobot dataset stores. You'll also see it called a trajectory, rollout, or demonstration. Episode 30 runs 470 timesteps, or 15.7 seconds, with the task "Put the red cube on top of the blue cube." Across all 383 episodes, you get 143,799 timesteps.
Keep it honest. The full definition of proprioception includes velocities, torques, and end-effector pose, but an SO-100 records joints and cameras. In this dataset observation.state holds the six joint positions, and there's no end-effector pose, no force, and no depth. Bigger rigs add those, but this dataset doesn't have them.

Where do episodes come from, and how are they stored?

Every episode in this dataset was recorded by teleoperation. Two arms, the SO-100 or SO-101 family. This is leader-follower teleoperation: a person moves the leader arm by hand, the follower arm mirrors it joint-for-joint, and the recorder saves both cameras and both sets of joint values until the task is done.
Teleoperation on the real robot gives you demonstrations on the exact arm and cameras the policy will run on. It's also expensive data, because every second is someone's time.
The other source is a policy rollout. Same arm, same cameras, no leader arm. A trained policy reads the images and joint positions and sends the actions. The recorder writes the same two images and twelve numbers. Teams also let a person take over when the policy gets stuck, and those takeovers are flagged in the recording. One dataset can hold demonstrations, autonomous attempts, and human corrections, so the format must support all three.
LeRobot is Hugging Face's open-source robot learning library, and LeRobotDataset is its format. On disk, a dataset is three folders:
  • meta/ is the map: the schema, the task instructions, and an episodes table with one row per episode.
  • data/ is the numbers: Parquet files with one row per timestep (state, action, timestamp).
  • videos/ is the pixels: MP4 files, one stream per camera.
In LeRobotDataset v3, an episode is a slice of shared Parquet and MP4 files, located through offsets in meta/.
Good to know. Is an episode a file? No. In the stacking set, all 56 episodes share one Parquet file and one MP4 per camera. The episodes table holds the offsets. Version 3 of the format works this way because the older layout, one file per episode per camera, is fine at fifty episodes and painful at a million. Three of the seven sources here were published in the older layout and converted.
Which brings the problem. You can't inspect this data by opening files. You need something that reads the pointers.

What does one episode look like in Voxel51?

Voxel51 reads those pointers. When you load a LeRobot dataset, each episode becomes one sample, and one tile in the grid is one episode.
The sample holds what you'd filter and sort on: episode index, task, number of frames, duration, frame rate, robot type. For episode 30, that's 470 frames, 15.7 seconds, and the task string above. It also holds a reference back to the source files. The video and the per-timestep numbers stay in the MP4 and Parquet files, and when you open an episode, the App reads the frames through that reference.
Episode 30 of svla_so100_stacking open in Voxel51 with both cameras, joint plots, and episode statistics, followed by a tasks filter and a labeled error demo from another source.
Each LeRobot episode is one Voxel51 sample, with filterable fields and synchronized cameras and plots.
Open episode 30 and both cameras play together, with the state and action plots underneath on one timeline. Every row in the grid opens into the full recording.
For this dataset, I added three fields on top of what the loader creates:
  • source says which of the seven sets an episode came from.
  • uniqueness is a per-episode score computed from embeddings. Higher means less like the rest of the dataset.
  • nearest_episode and nearest_episode_distance give, within one source, the closest other episode and how far away it is.

What's in the 383-episode LeRobot dataset?

Voxel51 histograms of the source and task fields for the 383-episode SO-100 and SO-101 Tabletop dataset, showing seven sources and ten task strings.
Seven sources and ten task strings across 383 LeRobot episodes.
Seven sources, all on the SO-100 and SO-101 arm family. Four are official LeRobot sets, and three were recorded by community members and published on the Hub. Every task is tabletop: pick, place, stack, or sort. All of it is Apache 2.0.
I merged them and left the recordings as published, flaws included.
Look at the camera names. The sources use top and wrist, up and side, and image and image2. Seven sources, three naming schemes. Any code that reads cameras by name has to handle all three.
To see the whole thing at once, embed the clips and plot them. The embeddings come from a Qwen3-VL embedding model (qwen3-vl-embedding-2b) run over five-second windows of video, one camera at a time. Similar clips land next to each other, so the plot forms islands, roughly one per source and camera (some wrist cameras overlap). Color the points by source, then click an island, lasso it, and see the clips in it.
Voxel51 embeddings panel with 2,290 clip embeddings in separate islands, and a lasso selection that shows 20 episodes from one source in the grid.
Qwen3-VL clip embeddings form roughly one island per source and camera.
Each of the five problems below lives somewhere on that plot.

Which tool finds which problem?

The three ideas behind today's robot models turn out to be the inspection tools. Text gives you embeddings, similarity, and search in words. Action gives you plots of state and action on every episode. Label gives you a detector you can point at the frames. The five problems below used a plain filter plus text and embeddings:
ProblemFound with
1. Aborted takes and bad metadataA duration filter, then a uniqueness sort
2. Redundant demonstrationsDistance to the nearest episode
3. Coverage gapsThe plot, then a search per color
4. Visual outliersThe plot, then similarity search
5. Finding by descriptionSearch in words
The five data problems in the SO-100 and SO-101 Tabletop dataset, and the Voxel51 tool that found each one.

Problem 1: Which episodes are broken before anything else?

Voxel51 duration histogram for 383 episodes, then a saved view filtered to under 5 seconds that returns six short episodes.
A duration filter under 5 seconds returns six aborted LeRobot episodes.
Start with the cheapest signal you have: how long each episode is.
Drag duration under 5 seconds, or open the saved view Episodes under 5 seconds, and six episodes come back:
  • svla_so100_stacking episodes 20 (1 frame), 37 (14 frames), 38 (28 frames), and 55 (141 frames)
  • svla_so100_sorting episode 23 (51 frames)
  • ityl_recording2 episode 2 (1 frame)
Open episode 20 or episode 2 of ityl_recording2 and you get a single black frame. These are aborted takes.
Now clear the filter and sort all 383 episodes by uniqueness, descending. The same six fill ranks 1, 2, 3, 4, 6, and 7. Rank 5 is svla_so100_stacking episode 39, a full-length episode, so the score isn't a perfect aborted-take detector. But a length filter and an embedding score landed on almost the same recordings, and neither was told what an aborted take is.
While you're here, open the task filter in the sidebar. There are ten distinct task strings. "Pick up the cube and place it in the box." (50 episodes) sits beside "pick-place-cube" (50 episodes), and one reads "Puttingg red block on blue one" (45 episodes). In robot_type, 333 episodes say so100 and 50 say so100_follower. A language-conditioned policy receives each of those strings as a different instruction.

Problem 2: Did someone record the same demonstration twice?

Saved view of 52 svla_so100_sorting episodes sorted by nearest_episode_distance, then episodes 40 and 41 opened to show matching starting cube positions.
Episodes 40 and 41 of svla_so100_sorting start with the cubes in the same places.
A person recording fifty demonstrations can easily put the cube back where it was.
The saved view Near-duplicate episodes (sorting) holds the 52 sorting episodes, sorted by nearest_episode_distance, smallest first. The top of the list pairs up: 40 and 41, 44 and 46, 0 and 3, 48 and 51, each listing its partner in nearest_episode. Open 40 and 41 at their first frame, and the red and blue cubes start in the same places.
Keep it honest. uniqueness doesn't find these pairs. Sort the whole dataset by uniqueness (ascending), and 19 of the 20 least unique episodes come from ityl_recording2. That points to the most repetitive source. Finding the pairs needed a comparison inside one source, which is why nearest_episode_distance exists.

Problem 3: What does "pick-place-cube" leave out?

Grid of 50 cscsxx_pick_place_cube episodes in Voxel51, all with the task pick-place-cube, showing yellow, green, and other cube colors in runs.
One task string, pick-place-cube, covers five cube colors.
The task says pick-place-cube. It doesn't say which cube.
The saved view One task, five cube colors holds 50 episodes from cscsxx_pick_place_cube, all with that exact task string. Scroll the grid, and you see five cube colors, recorded in runs of about ten episodes each. A zero-shot count gave red 10, orange 10, yellow 9, green 11, and blue 10.
Nothing in the metadata records the color. If your policy has to handle a yellow cube, you have nine demonstrations of it.
The merged dataset has the same skew from another direction. The official pick-and-place set showed a red cube in every frame checked, so after merging, pick-and-place has about 60 red-cube episodes against about 10 for each other color.
Keep it honest. Those color counts are zero-shot predictions from a model. I checked 30 of the 50 episodes by eye. Treat the counts as approximate.

Problem 4: What's in the frame that has nothing to do with the task?

Voxel51 embeddings plot colored by source with cscsxx_pick_place_cube windows selected, then episode 10 playing from the image2 camera with the room behind the table in view.
Windows at the edge of the image2 island lead to episodes with people behind the table.
On the plot, go to the island for cscsxx_pick_place_cube, camera image2, and select the windows at its edge. These are windows with people standing or sitting behind the table. Run a similarity search from one of them, and you get more windows from the same camera with people in them. In a local pass over the contact sheets, a person was visible in windows from 14 episodes of that source: 5, 10, 11, 12, 13, 15, 16, 26, 39, 40, 41, 42, 45, and 48.
Then load the saved view, Hand in frame, and play the first few seconds of the top camera. It's svla_so100_stacking episode 54, and a human hand reaches into the shot.
The per-episode uniqueness score doesn't catch any of this. The hand episode ranks 70th of 383, and the bystander episodes rank anywhere from 58th to 295th. One hand in one five-second window disappears when you average the whole episode. So you look at windows, not episodes.

Problem 5: Can you find it by describing it?

Voxel51 text search for the gripper is closed and holding nothing, returning 25 windows that mostly show the gripper holding an orange cube.
A text search for an empty gripper returns windows where the gripper holds a cube.
If you can describe it, you can search for it. Mostly.
Three searches on the plot:
  • "a person visible in the background" returns image2 windows from cscsxx_pick_place_cube with people behind the table. In a local run, the top 8 were all correct.
  • "a red cube stacked on top of a blue cube" returns wrist-camera windows from svla_so100_stacking with the red cube on the blue one. Top 8 again, all correct.
  • "the gripper is closed and holding nothing" returns windows where the gripper is holding a cube.
The third one fails, and it fails in a useful way. In this test, search matched what's in the picture and missed what's absent, so don't use it to prove an absence.

What are the five LeRobot dataset quality checks?

Five habits you can run on any LeRobot dataset:
  1. Sort by duration and open everything that's only a few seconds long.
  2. Read every distinct task string, camera name, and robot type across your sources.
  3. Inside each source, look at the pairs of episodes that are closest to each other.
  4. Open the embeddings plot and explain every island to yourself. If you can't explain one, open it.
  5. Search in words, both for what shouldn't be there and for what the task says should be.
All of these come before you spend a GPU hour.
Keep it honest. These checks look at what the cameras saw and at episode metadata. They don't cover dropped frames, joints that jump, or cameras and joints out of sync. Those need their own checks. This post also doesn't use a detector as a probe, which is a separate experiment.

Try it yourself

Sign up, then start with the saved views. Four of them open directly onto the problems above.

Frequently asked questions

Headshot of Harpreet Sahota
Hacker-in-Residence
Harpreet is Hacker-in-Residence at Voxel51, where he turns cutting-edge AI ideas into open-source prototypes and demos that push the boundaries of deep learning. From building tools that inspire to creating content that educates, Harpreet helps the AI community level up—one wild idea at a time.
See all articles by Harpreet Sahota

Try Voxel51 for yourself