Object tracking is a video annotation and prediction task that follows the same object across frames, giving it a consistent identity over time so its motion can be analyzed. Each tracked object keeps one ID as its box or mask moves frame to frame.
Object tracking extends detection into time. Instead of labeling each frame independently, you link an object's annotations across frames so that "car 3" in frame 1 is the same "car 3" in frame 100. The output is a set of tracks, each a consistent identity with a trajectory of boxes or masks through the video.
As an annotation task it is far more than per-frame labeling: the hard part is maintaining identity through occlusion, when objects cross, disappear behind something, and reappear.
Key takeaways
Tracking assigns a persistent identity to an object across frames, not just per-frame boxes.
The output is a trajectory per object, enabling motion, speed, and behavior analysis.
The defining challenge is identity through occlusion and re-identification after an object reappears.
What object tracking provides
Common tracking approaches and the aids that speed annotation.
Common tracking approaches and the aids that speed annotation.
Approach
What it means
Tracking-by-detection
Detect per frame, then link detections into tracks
Single vs multi-object
Follow one target, or many at once
2D vs 3D
Tracks on the image plane, or in 3D space for LiDAR scenes
Annotation aids
Interpolation between keyframes and model-assisted propagation
How it works
Annotators label an object on keyframes and propagate or interpolate its box across the frames between, then fix where identity breaks. In FiftyOne, video labels carry per-object IDs across frames, so you can play tracks, spot ID switches, and compare predicted tracks against ground truth.
Why it matters
Tracking is what turns "there is a car" into "this car is accelerating toward the crosswalk," the basis of motion understanding for AV, robotics, and analytics. The metric that matters is not per-frame accuracy but identity. A tracker can detect every object in every frame and still be useless if it swaps IDs, because an ID switch when two people cross ruins the trajectory even though detection was perfect. That is why tracking is scored with identity-aware metrics like MOTA and IDF1, and why the expensive annotation work is the occlusion and crossing cases, not the easy stretches.
Frequently asked questions
What is the difference between object detection and object tracking?
Detection finds objects in a single frame. Tracking links them across frames with a consistent identity.
What is an ID switch?
When a tracker mistakenly swaps the identities of two objects, breaking their trajectories.
How is tracking annotated efficiently?
With keyframes plus interpolation and model-assisted propagation, correcting where identity breaks.