Multi-object tracking
Multi-object tracking, or MOT, is the task of detecting multiple objects in a video and maintaining a consistent identity for each one across frames. It produces trajectories, linking detections over time so each object keeps the same ID.
In this article
Building visual or physical AI?
Let's talk.
What is multi-object tracking?
Multi-object tracking follows many objects at once through a video, assigning each a stable identity that persists frame to frame. The output is a set of trajectories, one per object, so that the same car or person keeps the same ID as it moves, is briefly occluded, or crosses paths with others.
The hard part is the data association: correctly linking each new detection to the right existing track.
Key takeaways
- MOT tracks many objects at once and keeps each ID consistent.
- Its output is a trajectory per object across the video.
- The core challenge is associating detections to the right tracks.
How it works
The dominant approach, tracking-by-detection, runs an object detector on each frame and then links detections across frames into tracks. Association uses cues like predicted motion, spatial overlap, and appearance embeddings, and re-identification helps recover tracks after occlusion. Optical flow and motion models help predict where each object should appear next, reducing ID switches.
Before you score a tracker, watch where identities could switch
Multi-object tracking scores like MOTA, IDF1, and HOTA fold every mistake into one number. Two trackers with the same score can fail in very different ways, and watching the clips is how you find out which. Many of the hardest moments come from the scene itself. The quiet problems:
- an identity switch when two people cross paths
- a track that ends when a person walks behind a parked car and restarts with a new ID
- ground truth tracks with gaps where an annotator skipped frames
In Voxel51, MEVA 500 has 500 short security-camera clips of staged everyday activities, with per-frame person and vehicle tracks and temporal activity detections.
MEVA 500
500 clips
Short security-camera clips of staged everyday activities, with per-frame person and vehicle tracks and temporal activity detections.
Try this: Find a clip where two people cross paths and check whether each one keeps the same track through the crossing.
Open MEVA 500 in Voxel51
Free with a work or .edu email.
Why it matters
Counting, behavior analysis, and autonomous perception all depend on knowing not just what objects are present but which is which over time. MOT is a core video capability in visual AI, and its reliance on detection, re-identification, and motion cues ties several tasks together.
Frequently asked questions
What is tracking-by-detection?
An approach that detects objects in each frame and then links those detections across frames into consistent tracks.
What makes multi-object tracking hard?
Correctly associating detections when objects occlude one another, look similar, or leave and re-enter the frame, which can otherwise cause identity switches.
Related terms
Director of Growth
Jesse Mostipak is the Director of Growth at Voxel51, where the work is helping humans find and trust what the brand knows, and teaching the Google knowledge graph and the LLMs answering on their behalf to do the same. That question, how knowledge gets built inside a system, is one Jesse has been chasing for years. Earlier versions of it ran through a New York City high school science classroom, data science and machine learning, and developer relations at Kaggle, Posit (formerly RStudio), and Baseten. The answer doesn't change much depending on whether the learner is a teenager, a software engineer, or a knowledge graph. Jesse holds a Master's in Education from CUNY Hunter College. LinkedIn
See all articles by Jesse Mostipak
Last updated November 11, 2026
Building visual or physical AI?
Let's talk.
© 2026 Voxel51 All Rights Reserved