Temporal detection

Temporal detection is about finding when something happens, not just whether it appears. Ordinary detection asks what is present in a single image, but many important things in the physical world are not moments, they are episodes that unfold over time. A pedestrian crossing the street, a robot grasping and lifting an object, or a vehicle changing lanes each has a beginning and an end, and understanding it means locating that interval within a sequence. An event, in this sense, is a meaningful occurrence defined over a span of time rather than a single instant.

This temporal framing matters because a snapshot can be ambiguous in ways that a sequence is not. A single frame might show a hand near a cup, but only the surrounding frames reveal whether the hand is reaching for it, holding it, or setting it down. Temporal detection labels these intervals explicitly, marking where an event starts, where it ends, and what kind of event it is. This turns raw video and sensor streams into structured descriptions of behavior, which is what downstream systems need in order to reason about actions and interactions.

Temporal detection operates over a sequence and reasons about how the content changes across it. The goal is to segment the timeline into intervals and assign each interval a label describing the event it contains, along with its boundaries in time. Because events have fuzzy edges, deciding exactly when a grasp begins or a crossing ends often requires clear conventions, which makes consistent annotation guidelines important. When events involve specific objects, temporal detection is closely tied to tracking those objects across frames, so that an event can be associated with the entity that performs or undergoes it.

For physical AI, most of what a system needs to understand is behavior, and behavior lives in time. Temporal detection matters to anyone building perception or evaluation for video and sensor streams, because it is what lets a system recognize actions, interactions, and rare events rather than isolated snapshots. Well-defined event annotations are also foundational to evaluation, since knowing exactly when something happened is a prerequisite for checking whether a system responded correctly and on time.

Ordinary detection finds what is present in a single frame, while temporal detection finds when an event occurs across a sequence, marking its start and end. One reasons about space in an image, the other reasons about time in a stream.

An event is a meaningful occurrence that unfolds over an interval of time, such as a pedestrian crossing, a grasp, or a lane change. What matters is that it has a beginning and an end and carries meaning for the task at hand.

Events often have fuzzy edges, so it can be genuinely ambiguous when a grasp begins or a crossing ends. Consistent annotation conventions are needed so that different annotators mark boundaries the same way, which keeps the resulting labels reliable.

Label track, Time-series sensor data, Episode / trajectory / rollout, Long-horizon task, Multimodal sensor data.