Segment-level embedding

A segment-level embedding represents a piece of a larger item. Instead of one vector for an entire ten-minute robot episode or video, you embed its segments, a few seconds of manipulation, a lane change, a masked object region, so each becomes its own point in embedding space. The whole recording stays intact, but its contents become individually findable.

The right granularity depends on the question: item-level embeddings answer "which recordings are similar," while segment-level embeddings answer "where inside them did this kind of moment happen."

The data is divided into segments, by fixed windows, detected events, shots, or masks, and each segment is passed through an embedding model suited to its modality. The resulting vectors are indexed alongside references back to their parent item and time range, so a similarity hit points to the exact moment. Pooling segment embeddings back together can also produce an item-level summary.

Physical AI data is long and mostly uneventful, and the valuable parts are brief: the near-miss, the failed grasp, the rare interaction. Segment-level embeddings are how you retrieve those moments across thousands of hours without watching any of it, which makes them central to failure mining, scenario mining, and curation of long-form multimodal data.

A frame embedding represents a single instant, while a segment embedding summarizes a span, capturing motion and behavior that single frames miss.

Fixed-length windows are the simplest, while event detection or shot boundaries give more semantically meaningful segments. The choice should match what you want to retrieve.

Embedding, Episode embedding, Frame embedding, Video embedding, Instance segmentation.

Learn: Multimodal embeddings for physical AI

Try it: Using image embeddings in FiftyOne · FiftyOne Brain