Object tracking
Object tracking is a video annotation and prediction task that follows the same object across frames, giving it a consistent identity over time so its motion can be analyzed. Each tracked object keeps one ID as its box or mask moves frame to frame.
In this article
Building visual or physical AI?
Let's talk.
What is object tracking?
Object tracking extends detection into time. Instead of labeling each frame independently, you link an object's annotations across frames so that "car 3" in frame 1 is the same "car 3" in frame 100. The output is a set of tracks, each a consistent identity with a trajectory of boxes or masks through the video.
As an annotation task it is far more than per-frame labeling: the hard part is maintaining identity through occlusion, when objects cross, disappear behind something, and reappear.
Key takeaways
- Tracking assigns a persistent identity to an object across frames, not just per-frame boxes.
- The output is a trajectory per object, enabling motion, speed, and behavior analysis.
- The defining challenge is identity through occlusion and re-identification after an object reappears.
What object tracking provides
| Approach | What it means |
|---|---|
| Tracking-by-detection | Detect per frame, then link detections into tracks |
| Single vs multi-object | Follow one target, or many at once |
| 2D vs 3D | Tracks on the image plane, or in 3D space for LiDAR scenes |
| Annotation aids | Interpolation between keyframes and model-assisted propagation |
How it works
Annotators label an object on keyframes and propagate or interpolate its box across the frames between, then fix where identity breaks.
Why it matters
Tracking is what turns "there is a car" into "this car is accelerating toward the crosswalk," the basis of motion understanding for AV, robotics, and analytics. The metric that matters is not per-frame accuracy but identity. A tracker can detect every object in every frame and still be useless if it swaps IDs, because an ID switch when two people cross ruins the trajectory even though detection was perfect. That is why tracking is scored with identity-aware metrics like MOTA and IDF1, and why the expensive annotation work is the occlusion and crossing cases, not the easy stretches.
Frequently asked questions
What is the difference between object detection and object tracking?
Detection finds objects in a single frame. Tracking links them across frames with a consistent identity.
What is an ID switch?
When a tracker mistakenly swaps the identities of two objects, breaking their trajectories.
How is tracking annotated efficiently?
With keyframes plus interpolation and model-assisted propagation, correcting where identity breaks.
Related terms
Director of Growth
Jesse Mostipak is the Director of Growth at Voxel51, where the work is helping humans find and trust what the brand knows, and teaching the Google knowledge graph and the LLMs answering on their behalf to do the same. That question, how knowledge gets built inside a system, is one Jesse has been chasing for years. Earlier versions of it ran through a New York City high school science classroom, data science and machine learning, and developer relations at Kaggle, Posit (formerly RStudio), and Baseten. The answer doesn't change much depending on whether the learner is a teenager, a software engineer, or a knowledge graph. Jesse holds a Master's in Education from CUNY Hunter College. LinkedIn
See all articles by Jesse Mostipak
Last updated November 3, 2026
Building visual or physical AI?
Let's talk.
© 2026 Voxel51 All Rights Reserved