Robotics foundation models keep getting bigger, but they keep hitting the same wall. As Lightwheel puts it, models today all face the same fundamental constraint: the field lacks sufficient, diverse, and high-quality robot-usable data. EgoSuite-Open100K is one of the most substantial open answers to that shortage so far, and it turns out to pair remarkably well with a tool built for exactly this kind of multimodal exploration. We put the two together in a hands-on notebook, and this is what we found.
Key takeaways
EgoSuite-Open100K is Lightwheel's open egocentric human dataset, covering 100,000 hours across more than 15,000 tasks and scenes, with each episode stored as a single MCAP file.
Because MCAP embeds its own Protobuf schemas, FiftyOne decodes Lightwheel's custom pose and annotation message types with no .proto files, no SDK, and no ROS or Foxglove toolchain.
FiftyOne reads EgoSuite-Open100K MCAP files natively, playing stereo head-camera video, full-body and hand pose, and action segments back on one shared timeline without pre-extracting frames.
Capability fields derived from MCAP schemas, such as has_image, has_hand_pose, and has_body_pose, let you filter EgoSuite episodes by modality without opening a single file.
Motion-similarity embeddings computed over EgoSuite pose streams cluster episodes by how the person actually moved, making an egocentric robotics dataset searchable rather than just browsable.
What's inside EgoSuite-Open100K
Lightwheel describes it as the largest fully annotated open egocentric human dataset — 100,000 hours across 15,000+ tasks and scenes. But raw scale isn't the interesting part. Density is. These aren't scraped web videos. EgoSuite is a global-scale egocentric human data collection operation: trained operators wearing VR headsets, smart glasses, and exoskeletons perform tasks in real-world environments such as homes, factories, warehouses, public spaces, while capturing first-person video, hand tracking, depth maps, and action segmentation data.
What a single EgoSuite-Open100K MCAP episode contains, and how FiftyOne surfaces each stream.
What a single EgoSuite-Open100K MCAP episode contains, and how FiftyOne surfaces each stream.
Stream in the MCAP file
What it captures
How FiftyOne surfaces it
Stereo head-camera video
First-person left and right camera views
Plays back on the shared episode timeline, no frame pre-extraction
Full-body and hand pose
3D joint transforms over time
Decoded from embedded Protobuf descriptors, used for motion-similarity embeddings
Action segments
Frame-accurate semantic labels for what the operator is doing
Temporal labels you can filter, tag, and save as a view
Schema records
The message definitions for every stream in the file
Capability fields such as has_image, has_hand_pose, and has_body_pose
And diversity is the deliberate design goal. Lightwheel's own thesis is that what you want is humans performing thousands of different activities across hundreds of different environments, not the same pick-and-place repeated a million times. You feel that the moment you open a handful of episodes: aligning chairs, adjusting a salt grinder, affixing shipping labels, sorting a desktop. Each one a single self-describing MCAP file carrying stereo head-camera video, full-body and hand pose as 3D joint transforms, and frame-accurate action segments, all synchronized on one clock.
Lightwheel: infrastructure for physical AI
Lightwheel is a robotics infrastructure company that develops simulation, synthetic data, evaluation, and deployment systems for training and scaling physical AI robots in real-world environments. Its data engine has three layers: SimReady assets (physically accurate 3D scenes that form the world foundation for training in simulation), EgoSuite (globally scalable egocentric human data), and RoboFinals (an industrial-grade simulation evaluation platform built to challenge frontier vision-language-action (VLA) and world models).
EgoSuite is the middle layer and their most distinctive bet. By their published figures, the platform has delivered over 300,000 hours of egocentric data, with a production rate exceeding 20,000 hours per week, spanning multiple countries and environment types. Open100K is the open slice of that pipeline.
FiftyOne: native MCAP ingestion, no pipeline required
FiftyOne is Voxel51's open-source platform for annotating, visualizing, curating, and improving machine learning datasets, the layer where you actually look at your data, filter it, compute embeddings over it, and turn raw recordings into curated training sets. For images and video that's long been smooth. For robotics logs it's traditionally meant a ROS or Foxglove toolchain and a lossy extraction step before you can see anything.
That's what makes this pairing click: FiftyOne now reads MCAP natively. It natively ingests MCAP, the default recording format for ROS 2, and camera streams, LiDAR point clouds, and numeric time-series sensors along with robot state all play back in sync on a single shared timeline. Crucially, it works directly with encoded stream media and infers frame-level data from it, so you do not have to pre-extract frames. On top of playback, the platform lets you tag arbitrary intervals within episodes and turn them into persistent, queryable assets, with labels that persist as tracks across an entire episode rather than isolated frame-by-frame annotations.
For a dataset that ships every episode as MCAP, that's the difference between "stand up a pipeline" and "just open it."
How to explore EgoSuite-Open100K MCAP files in FiftyOne
We built a self-contained, reproducible notebook that goes from a raw, gated dataset to a curated, queryable, action-labeled collection, entirely inside FiftyOne. In a handful of cells it:
Downloads a diverse subset: ~50 distinct manipulation tasks, selected one-per-task so the sample spans many activities rather than repeating one.
Opens the MCAP files natively: stereo head-camera video plus full-body and hand pose, scrubbing together on a single timeline, no ROS or Foxglove in the loop.
Derives capability fields from schemas, not filenames: has_image, has_hand_pose, has_body_pose — so you can filter to episodes that genuinely contain a modality without opening them.
Builds motion-similarity embeddings: FiftyOne decodes Lightwheel's custom pose message types straight from the Protobuf descriptors embedded in each MCAP file, with no .proto files and no SDK. The embedding clusters episodes by how the person actually moved, and the Embeddings panel plus similarity search let you explore and mine those patterns.
Turns action segmentation into filterable, taggable labels: the semantic segments become temporal labels you can filter by action (narrow 50 episodes to just the ones containing a "pick"), tag, and save as a view.
The throughline is that self-describing decode. Reading proprietary pose and annotation schemas out of the file itself, with no external definitions required, is the clearest demonstration of why MCAP matters, and it's what lets a general-purpose data tool such as FiftyOne turn an egocentric robotics firehose into something searchable, curatable, and ready to build on.
FAQ
EgoSuite-Open100K is Lightwheel's open egocentric human dataset, spanning 100,000 hours across more than 15,000 tasks and scenes. Trained operators wearing virtual reality headsets, smart glasses, and exoskeletons perform everyday tasks in homes, factories, warehouses, and public spaces while capturing first-person video, hand tracking, depth maps, and action segmentation data.
You open them directly in FiftyOne, which reads MCAP natively. Stereo head-camera video, full-body and hand pose, and action segments all play back on a single shared timeline with no ROS or Foxglove toolchain in the loop.
No. Each MCAP file embeds the Protobuf descriptors for its own message types, so FiftyOne decodes Lightwheel's custom pose and annotation schemas straight from the file with no external definitions and no SDK.
FiftyOne derives capability fields from the MCAP schemas rather than from filenames, so fields such as has_image, has_hand_pose, and has_body_pose let you filter to episodes that genuinely contain a modality without opening them.
Lightwheel's data engine has three layers: SimReady assets, which are physically accurate 3D scenes for training in simulation, EgoSuite, which is globally scalable egocentric human data, and RoboFinals, an industrial-grade simulation evaluation platform for frontier vision-language-action and world models.
Next steps and resources
Want to try it yourself? Everything you need is below.
