What is an episode embedding?
Robot learning datasets are collections of episodes, recordings of a robot attempting a task, containing synchronized observations, actions, and outcomes. An episode embedding, also called a trajectory embedding, compresses one whole episode into a vector, placing successful pick-and-place attempts near each other, hesitant grasps in their own region, and collisions somewhere distinctive.
It is the episode-level granularity of embedding, complementing segment-level embeddings that expose the moments within.
Key takeaways
- Episode embeddings represent entire robot rollouts as single vectors.
- Similar behaviors and outcomes cluster together.
- They pair with segment-level embeddings for within-episode search.
How it works
The episode's streams, video, proprioception, actions, are encoded over time by sequence or video models, and the per-window features are pooled into one vector per episode. Clustering these vectors groups a dataset by behavior, retrieval finds episodes like a query episode, and outliers surface unusual attempts worth review. Since pooling dilutes brief moments, teams typically index segments alongside episodes.
Why it matters
Datasets like Open X-Embodiment aggregate over a million episodes, far beyond manual review, and episode embeddings are how such libraries become organized: deduplicate near-identical demonstrations, balance behavior coverage, find every episode resembling a failure. That curation directly shapes what imitation-learned policies learn.
Frequently asked questions
What is the difference between episode-level and segment-level embeddings?
Episode embeddings summarize whole recordings and answer which episodes are similar. Segment embeddings expose the moments within them and answer where something happened.
Do episode embeddings include actions or just observations?
Either. Embedding observations captures what the robot saw, while including actions captures what it did, and behavior-focused curation usually wants both.
Related terms
Go deeper