What is multi-object tracking?
Multi-object tracking follows many objects at once through a video, assigning each a stable identity that persists frame to frame. The output is a set of trajectories, one per object, so that the same car or person keeps the same ID as it moves, is briefly occluded, or crosses paths with others.
The hard part is the data association: correctly linking each new detection to the right existing track.
Key takeaways
- MOT tracks many objects at once and keeps each ID consistent.
- Its output is a trajectory per object across the video.
- The core challenge is associating detections to the right tracks.
How it works
The dominant approach, tracking-by-detection, runs an object detector on each frame and then links detections across frames into tracks. Association uses cues like predicted motion, spatial overlap, and appearance embeddings, and re-identification helps recover tracks after occlusion. Optical flow and motion models help predict where each object should appear next, reducing ID switches.
Why it matters
Counting, behavior analysis, and autonomous perception all depend on knowing not just what objects are present but which is which over time. MOT is a core video capability in visual AI, and its reliance on detection, re-identification, and motion cues ties several tasks together.
Frequently asked questions
What is tracking-by-detection?
An approach that detects objects in each frame and then links those detections across frames into consistent tracks.
What makes multi-object tracking hard?
Correctly associating detections when objects occlude one another, look similar, or leave and re-enter the frame, which can otherwise cause identity switches.
Related terms