Robotics is having its "ImageNet moment," and the dataset driving a lot of it is Open X-Embodiment (OXE). But a million robot trajectories are only as useful as your ability to look at them, filter them, and understand what's actually inside. That's where FiftyOne comes in. In this post I'll walk through a small, self-contained notebook that pulls an OXE dataset into FiftyOne as browsable, groupable, searchable video — and turns a pile of .parquet and .mp4 shards into something you can explore in your browser in minutes.
FiftyOne app showing the oxe_fanuc_demo dataset as a grid of 830 robot manipulation samples, with a yellow FANUC arm above a green work surface and the sidebar listing task, episode_index, and camera_view fields.
Key takeaways
Open X-Embodiment (OXE) pools 60 datasets from 34 research labs into over 1 million real robot trajectories spanning 22 robot embodiments, 527 skills, and roughly 160,000 tasks.
Because OXE was assembled from independently collected datasets, camera setups, state and action spaces, resolutions, and metadata quality vary from sub-dataset to sub-dataset, which makes visual inspection a prerequisite to training.
FiftyOne maps Open X-Embodiment structure directly onto its own primitives: multi-camera episodes become grouped datasets, per-frame trajectories become queryable frame-level fields, and natural-language task strings become first-class fields.
The demo notebook loads the berkeley_fanuc_manipulation OXE dataset, 415 episodes across two cameras with 32 language-annotated tasks, from raw Hub shards into a browsable FiftyOne dataset.
CLIP embeddings projected with UMAP inside the FiftyOne Brain reveal how Open X-Embodiment scenes, tasks, and trajectory phases cluster, turning a pile of parquet and mp4 shards into an interactive map of the dataset.
What is Open X-Embodiment? A million trajectories, 22 Robots, one dataset
OXE is the largest open-source real-robot dataset assembled to date. It contains over 1 million real robot trajectories spanning 22 robot embodiments — from single arms to bi-manual robots and quadrupeds — built by pooling 60 existing datasets from 34 research labs into one consistent format. Across all of that it demonstrates 527 skills and roughly 160,000 tasks, each segmented and annotated with natural-language instructions.
The reason this matters isn't just scale — it's diversity. The whole premise of OXE (and the RT-X models trained on it) is that a policy trained across many robots, scenes, and tasks generalizes better than one trained on a single embodiment, even when you evaluate it on that same single embodiment. Diversity is the feature.
But diversity is also the challenge. Because OXE was pooled from 60 independently collected datasets, the sub-datasets don't share a fixed structure: camera setups differ, state and action spaces differ, resolutions and framerates differ, and metadata quality varies lab to lab. Before you train anything on this data, you really want to see what you've got. And "seeing" a million videos with per-frame sensor streams is not a matplotlib job.
FiftyOne: eyes for your dataset
FiftyOne is an open-source toolkit from Voxel51 for annotating, curating and understanding visual datasets. It gives you a Python API and a browser-based app for loading images, video, and multimodal sensor data; slicing and filtering it with expressive queries; computing embeddings to reveal hidden structure; and searching by similarity or natural language. It's built for exactly the messy, multimodal, real-world data that physical-AI and autonomous-systems teams wrestle with — the kind where the hard part isn't the model, it's knowing what's in your data and finding the samples that matter.
From Open X-Embodiment data to FiftyOne workflows
OXE and FiftyOne fit together almost too well, because OXE surfaces every problem FiftyOne is designed to solve:
Multi-camera episodes map cleanly onto FiftyOne's grouped datasets — each episode becomes a group, and each camera view becomes a slice you can flip between in the app.
Per-frame trajectories (joint states, end-effector actions, timestamps) become frame-level fields you can query and filter, not opaque arrays buried in a parquet file.
Natural-language task strings become first-class fields, so "show me every episode with a Rubik’s cube" is one line of code.
Within-dataset structure — even a single OXE sub-dataset has structure worth auditing, and FiftyOne's embeddings and similarity tools surface it: spot near-duplicate frames, find visual outliers, and see how scenes cluster by task.
In short: OXE gives you the diversity; FiftyOne gives you the eyes to make sense of it.
How Open X-Embodiment data structures map onto FiftyOne primitives.
How Open X-Embodiment data structures map onto FiftyOne primitives.
Open X-Embodiment data
FiftyOne primitive
What it unlocks
Multi-camera episodes
Grouped datasets, one slice per camera
Flip between scene and wrist views at the same moment
Query and filter sensor streams instead of unpacking arrays from parquet
Natural-language task strings
Sample-level fields
One-line queries like "show me every episode with a Rubik's cube"
Within-dataset visual structure
Embeddings and similarity indexes
Spot near-duplicate frames, find visual outliers, and see scenes cluster by task
From shards to insight: the berkeley_fanuc_manipulation demo
We use the berkeley_fanuc_manipulation dataset in our demo notebook: two cameras (a scene view and a wrist view), 415 episodes, and 32 language-annotated tasks — small enough to run fast, rich enough to be interesting. Once it's loaded into FiftyOne, the demo lights up:
Multi-camera playback. Scrub through episodes as video and flip between the scene camera and the wrist camera for the same moment using the group-slice dropdown. Two synchronized views of every trajectory, right in the browser.
Task and episode filtering. List all 32 tasks with counts, then isolate a single task or episode with one-line queries — the fast way to answer "what's actually in here, and show me just the drawer-opening runs."
Embeddings visualization.Compute CLIP embeddings on the frames and project them to 2D with UMAP inside FiftyOne's Brain. Color the points by task and watch the data cluster — scenes, tasks, and trajectory phases separate visually, giving you an instant map of the whole dataset's structure.
Similarity search. Index the frames and sort by visual similarity to any query frame — the "find me more moments that look like this one" workflow that makes curation fast.
Go from raw Hub shards to an interactive, searchable, embedding-mapped robot dataset in a single notebook.
Next steps with Open X-Embodiment and FiftyOne
Want to take it further? A few natural directions:
Go multi-embodiment. Re-run the validated pipeline on other OXE datasets (taco_play, bridge, fractal) and compute embeddings across all of them together — the cross-embodiment scene-diversity plot is the single most compelling visual you can produce, and it directly illustrates why OXE works.
Bring your own model. Run a VLA or perception model over the frames and use FiftyOne's evaluation views to find failure episodes.
Open X-Embodiment (OXE) is the largest open-source real-robot dataset assembled to date, containing over 1 million real robot trajectories across 22 robot embodiments. It was built by pooling 60 existing datasets from 34 research labs into one consistent format, covering 527 skills and roughly 160,000 tasks, each segmented and annotated with natural-language instructions.
You point FiftyOne at the OXE dataset's Hub shards and let it parse the parquet and mp4 files into samples. The demo notebook does this for berkeley_fanuc_manipulation, creating a grouped dataset where each episode is a group and each camera view is a slice.
Because OXE was pooled from 60 independently collected datasets, the sub-datasets do not share a fixed structure. Camera setups differ, state and action spaces differ, resolutions and framerates differ, and metadata quality varies lab to lab, so you need to see what you have before you train on it.
Yes. FiftyOne's grouped datasets make each episode a group and each camera view a slice, so you can scrub an episode as video and flip between the scene camera and the wrist camera for the same moment using the group-slice dropdown.
Natural-language task strings load as first-class fields, so filtering to a single task or episode is a one-line query. You can also list all 32 tasks with counts to see what is in the dataset before you filter.
