Label track

A label track is the thread that connects all the appearances of a single object across a sequence of frames. When you annotate a video or a sensor stream, you could draw a box around a car in every frame independently, but that alone would not tell a system that it is the same car each time. A label track adds exactly that missing information, giving the object a consistent identity so that its box in frame one, frame two, and frame three are all understood as belonging to one entity moving through the scene.

This continuity is what makes tracks so useful. With a track, you can measure how an object moves, how fast it goes, and how it interacts with others, because you know which observations belong together over time. Tracks apply to annotations of many kinds, from 2D boxes on images to 3D cuboids in point clouds, and they are essential wherever behavior over time is what matters. Maintaining a stable identity through occlusions, when an object briefly disappears behind another, is one of the trickier parts of building good tracks.

Building a label track means associating detections of the same object across consecutive frames and assigning them a shared identifier. The core challenge is data association, deciding which box in the next frame corresponds to which object in the current one, especially when objects are close together, moving quickly, or temporarily hidden. A well-formed track preserves identity through these moments, so the object does not lose its label when it reappears from behind an occlusion. The result is a per-object timeline that records where the object was and how it behaved throughout the sequence.

Label tracks are foundational for any physical AI system that needs to understand motion, since knowing that an object persists and where it goes is often more important than spotting it in a single frame. They matter to anyone producing training data or evaluating perception for video and sensor streams, because tracking quality directly shapes what a model can learn about behavior and how faithfully its performance can be measured. Consistent identities across time are what turn a set of isolated detections into a coherent story of what happened.

A per-frame label annotates an object in a single frame without connecting it to other frames. A label track links those annotations across frames under one identity, so the system knows they all refer to the same object moving through the sequence.

When an object passes behind another and reappears, a system has to recognize it as the same entity rather than a new one. Fast motion, similar-looking objects, and long occlusions all make this association difficult, and mistakes cause identities to be swapped or lost.

They are used to teach and evaluate systems that follow objects over time, enabling measurement of motion, speed, and interactions. They apply to both 2D annotations on images and 3D annotations such as cuboids in point clouds.

Temporal detection / event, 3D bounding box / cuboid, Time-series sensor data, Episode / trajectory / rollout, Multimodal sensor data.