HubEgo-100K: 338 Episodes of Real Skilled Work, Explorable in Your Browser
•
6 min read
Author

Hacker-in-Residence
Harpreet is Hacker-in-Residence at Voxel51, where he turns cutting-edge AI ideas into open-source prototypes and demos that push the boundaries of deep learning. From building tools that inspire to creating content that educates, Harpreet helps the AI community level up—one wild idea at a time.
See all articles by Harpreet Sahota
In this article
Key takeawaysWhat does one HubEgo-100K episode look like?Step 1: Start with the hard onesStep 2: Ask who has both hands busyStep 3: Combine the filters, then change the layoutStep 4: Find where the hands leave the frameStep 5: Check the annotator’s workStep 6: Split by place, skill, and phoneHow representative is this sample?Try it yourselfFrequently asked questions
In this article
Key takeawaysWhat does one HubEgo-100K episode look like?Step 1: Start with the hard onesStep 2: Ask who has both hands busyStep 3: Combine the filters, then change the layoutStep 4: Find where the hands leave the frameStep 5: Check the annotator’s workStep 6: Split by place, skill, and phoneHow representative is this sample?Try it yourselfFrequently asked questions
Try Voxel51 for yourself
The 338-episode HubEgo-100K sample in the Voxel51 samples grid, with the embeddings panel open.
A 338-episode sample of HubEgo-100K, first-person video of nail technicians, mechanics, warehouse staff, and carpenters at work, with 11 saved views you can open from a link.
A robot policy knows the rooms it was trained in.
The HubEgo-100K paper points to a DROID collection effort that took 12 months, 50 data collectors, and 564 scenes to reach 350 hours. It also cites scaling studies showing that generalization depends on the number of training environments and objects, while extra demonstrations in the same environment stop helping. More rooms beat more repeats, and rooms are expensive for a robot to visit.
People already work in thousands of them. HubEgo-100K, from Hub Labs, is 10,124 hours of 1080p first-person (egocentric) video of skilled work, recorded by 12,227 contributors in their own workshops, salons, kitchens, and building sites. Every episode comes with a 100 Hz motion sensor stream and a task description, and the whole release is Creative Commons Attribution 4.0 (CC BY 4.0).
That is a lot of video to look at. So we built a 338-episode sample in Voxel51, added saved views for the questions you would ask first, and linked each one. Open the samples grid, sign in or sign up, and you’ll see real work.
Key takeaways
- HubEgo-100K v1.0 is 10,124 hours across 72,754 episodes from 12,227 contributors, covering 56 task types, 35 action families, and 11 setting types. The Voxel51 sample holds 338 of those episodes (5.04 hours, 42 task types, 305 contributors, 28 phone models).
- Each HubEgo-100K episode is one MCAP file: 1080p H.264 video at 30 frames per second (fps) on
/camera/main/video, plus accelerometer, gyroscope, magnetometer, device motion, heading, and barometer topics on the same clock. - HubEgo-100K annotation fields such as
difficulty,action_family,skill_group,environment,hands_share_both, andon_task_sharesit on every sample, so “hard tasks where both hands stay in frame” is a filter that returns 25 episodes. - 11 saved views of the HubEgo-100K sample are live, each with its own URL. Add
?view=<slug>to the dataset link to open one, and add&workspace=<name>to open it inside a saved panel layout. - HubEgo-100K annotations and hand-visibility shares are model-estimated and have not been checked by people. The sample leans short and easy, and the iPhone and Android accelerometer values use different units. All three are covered below.
What does one HubEgo-100K episode look like?
One HubEgo-100K episode in Voxel51: first-person video and inertial measurement unit (IMU) streams play on the same timeline.
Open one stocking episode and press play. A worker on a shop floor stocks automotive supplies and applies price tags, for 483 seconds, through a head-mounted phone. The image tile plays the footage, and a plot tile can chart
/imu/accel and /imu/gyro against the same timeline.Under the hood, each episode is a single MCAP with these topics:
/camera/main/video: H.264, 1920x1080, 30 fps/imu/accel,/imu/gyro,/imu/mag: about 100 Hz/imu/motion,/imu/heading,/imu/baro: device motion, heading, barometer/video/frames: one message per video frame, linking it to its source frame and capture time
The sample-level fields come from the dataset’s own catalog and annotation tables.
task, category, environment, skill_group, action_family, difficulty, description, and task_instruction describe the work. hands_share, hands_share_both, and on_task_share measure how much of the episode shows a hand, both hands, and active work. device_model, os, and time_of_day describe the recording.Keep it honest. A vision-language model wrote the annotations, and models estimated the hand and on-task shares. Nobody has checked them by hand. Treat them as strong filters and spot-check what comes back, which the saved views make easy.
Step 1: Start with the hard ones
Filtering the HubEgo-100K sample to the 39 episodes labeled hard.
The paper reports that 34.5% of hours in the full release are labeled hard. Hard tasks are
difficulty == "hard": 39 episodes, longest first. It is the quickest way to see what the annotator means by hard, and whether you agree.Step 2: Ask who has both hands busy
The Two-handed work view keeps HubEgo-100K episodes where both hands are in frame more than 90% of the time.
Hand visibility is the obvious first filter for manipulation data. Two-handed work keeps episodes where both hands are in frame for more than 90% of sampled frames: 66 episodes, led by nail cleaning, nail buffing, and wire stripping at 99% or higher.
Good to know. Why does hand visibility matter for robot learning? The HubEgo-100K paper cites a study of Ego4D in which keeping only stable clips with a visible hand (975 of 3,976 hours) turned a negative co-training effect into a positive one. The
hands_share and hands_share_both fields let you apply that kind of filter yourself.Step 3: Combine the filters, then change the layout
Hard and two-handed: 25 HubEgo-100K episodes, with their embeddings highlighted.
Hard and two-handed is where it gets specific. Hard and two-handed returns 25 episodes where
difficulty is hard, and hands_share_both is above 0.7. That cutoff is looser than the 0.9 in Two-handed work on purpose: only 11 of the 39 hard episodes clear 0.9. Sort by hands_share_both and the top reads like a dexterity benchmark: filing and shaping an acrylic nail, filing nails with an electric drill, giving a puppy medication through an intravenous (IV) catheter with a syringe, applying gel polish with a brush.The same view also opens in a saved workspace with the samples grid, an embeddings panel with a color-by, and a task histogram, so you can see what the matches have in common and what matched. Do the 25 hard, two-handed episodes cluster together in the embedding space, or do they scatter across the task histogram? You find out by clicking.
Step 4: Find where the hands leave the frame
The hands often out of frame view surfaces HubEgo-100K episodes where a hand is visible less than 90% of the time.
The hand-tracking stress test is Hands often out of frame: 122 episodes where a hand is visible in under 90% of sampled frames, lowest first. The first one is a masonry wall chase cut with a hammer and chisel, with a hand in frame 40% of the time. The next is a watch being cleaned and polished with a cloth.
Step 5: Check the annotator’s work
Auditing the on_task_share estimate on HubEgo-100K episodes the model scored as mostly off-task.
Mostly off-task holds 41 episodes with
on_task_share under 0.8. It is also a free audit. The lowest episode scores 0.0 for “Inspect vehicle setup in repair workshop”, an episode the model says contains no active work. Open it and decide whether the model or the description is wrong.Step 6: Split by place, skill, and phone
Splitting the HubEgo-100K sample by environment with saved views such as Salon and personal care.
The remaining views slice by where and how the work happened:
- Workshops and repair benches: 107 episodes
- Tool use and technical manipulation: 102 episodes, the largest
skill_group - Salon and personal care: 76 episodes
- Morning, evening, and night recordings: 140 episodes recorded outside the afternoon
- Long recordings: the 4 episodes of 10 minutes or more, up to 30 minutes of sanding a wooden table
Android phones hold 13 episodes from Samsung devices, and it exists for a reason. The dataset card lists accelerometer values in m/s². In the MCAP files, the Android episodes match that, with a median magnitude near 9.8. The 325 iPhone episodes report in g, with a median near 1.0. We left the bytes untouched and added an
accel_units field (g or m/s^2) to every sample. If you train on /imu/accel, convert by os first.How representative is this sample?
Keep it honest. The sample is 338 of 72,754 episodes, and it skews short and easy. Nineteen episodes are a seeded random draw from the full release. The other 319 are a seeded random draw from its shortest episodes, so almost all of them run 29 to 36 seconds. The paper reports that 72.8% of hours in the full release sit in episodes longer than 10 minutes. Here, 4 episodes are that long.
Three effects follow. Only 39 episodes (11.5%) are hard, against 34.5% of hours in the full release. Salon, barber, and nail services alone make up 69 episodes (20.4%). The full release rebalances hours across task types so none dominates, and a sample drawn this way does not inherit that. Use the sample to learn the schema and test your filters. Use the full release for statistics.
Try it yourself
Every link opens in the Voxel51 app. Sign in or sign up and you land on the exact view.
| Saved view | Episodes | Link |
|---|---|---|
| All episodes | 338 | Samples grid |
| Hard tasks | 39 | Open view |
| Two-handed work | 66 | Open view |
| Hard and two-handed | 25 | Open view |
| Hands often out of frame | 122 | Open view |
| Workshops and repair benches | 107 | Open view |
| Tool use and technical manipulation | 102 | Open view |
| Long recordings | 4 | Open view |
| Mostly off-task | 41 | Open view |
| Salon and personal care | 76 | Open view |
| Android phones | 13 | Open view |
| Morning, evening and night recordings | 140 | Open view |
- Hard and two-handed in a saved workspace: samples, embeddings, and a task histogram
- One episode with video and inertial measurement unit (IMU) data
- HubEgo-100K on Hugging Face: the full release in LeRobot and MCAP formats (access form required)
- Paper: HubEgo-100K, arXiv:2611.21938
Credit: “HubEgo-100K, Hub Labs (Hub Data, Inc.), https://huggingface.co/datasets/HubLabs/HubEgo-100K, CC BY 4.0.” The Voxel51 copy modifies the release: it selects 338 episodes, muxes each episode’s video and sensor MCAP into one file, and attaches the annotation tables as sample fields.
Frequently asked questions
HubEgo-100K is a dataset of first-person (egocentric) video of skilled work from Hub Labs. Release 1 has 10,124 hours across 72,754 episodes, recorded with a head-mounted phone by 12,227 contributors in their own workplaces and homes. It ships in LeRobot and MCAP formats under CC BY 4.0.
Yes. The HubEgo-100K license is CC BY 4.0, which allows commercial use with credit. The credit line is in the Try it yourself section above.
Each HubEgo-100K episode includes accelerometer and gyroscope at about 100 Hz, plus magnetometer, device motion, heading, and barometer, all on the same clock as the 30 fps video. In the Voxel51 sample, they are the
/imu/accel, /imu/gyro, /imu/mag, /imu/motion, /imu/heading, and /imu/baro topics.In the HubEgo-100K MCAP files,
/imu/accel is in m/s² for the 13 Android episodes and in g for the 325 iPhone episodes. Each sample’s accel_units field says which one it is, and the Android phones view collects the Android episodes.Take the Voxel51 dataset’s samples URL and add
?view= followed by the saved view’s slug, which is the view name lowercased and hyphenated, with punctuation dropped. Add &workspace= and a saved workspace’s name to open the view inside that panel layout. A single sample opens at /sample/<id>.No. A vision-language model produced the
description, task_instruction, action_family, skill_group, and difficulty fields, and the model estimated hands_share, hands_share_both, and on_task_share. People have not checked them.Open the samples grid, click one episode, and press play.

Hacker-in-Residence
Harpreet is Hacker-in-Residence at Voxel51, where he turns cutting-edge AI ideas into open-source prototypes and demos that push the boundaries of deep learning. From building tools that inspire to creating content that educates, Harpreet helps the AI community level up—one wild idea at a time.
See all articles by Harpreet Sahota
Try Voxel51 for yourself
Related posts
Boost your visual and physical AI workflows.


