What is a spatiotemporal embedding?
Some content only makes sense as space and time together: a pedestrian stepping into a crosswalk, a robot arm approaching a grasp, traffic flowing through an intersection. A spatiotemporal embedding encodes such content so that both the spatial arrangement and its evolution shape the vector. Two moments with identical geometry but opposite motion embed differently.
It is the general term covering video embeddings, embedded sensor sequences, and motion-aware scene representations.
Key takeaways
- Spatiotemporal embeddings encode space and time jointly.
- Motion and unfolding behavior shape the vector, not just appearance.
- They underpin retrieval of dynamic events in video and sensor data.
How it works
Spatiotemporal encoders extend spatial architectures with time: video transformers attend across frames as well as within them, 3D convolutions slide over space-time volumes, and sequence models consume per-timestep features from any modality. The encoder pools its space-time features into one vector per window, and windows are indexed for retrieval like any embedding.
Why it matters
The costly events in physical AI are dynamic, cut-ins, falls, near-misses, failed manipulations, and static representations cannot distinguish them from their harmless lookalikes. Spatiotemporal embeddings make dynamics searchable, which is what lets teams mine fleets of recordings for behaviors rather than appearances.
Frequently asked questions
How does this differ from a video embedding?
Video embedding is the camera-specific case. Spatiotemporal embedding is the broader idea, applying equally to LiDAR sequences, telemetry, and multi-sensor streams.
What granularity do spatiotemporal embeddings use?
Windows of seconds are typical, embedded per window, since behavior unfolds over short spans and longer content is segmented first.
Related terms
Go deeper