Video annotation interpolation
Video annotation interpolation is a labeling technique that fills in an object's annotations on the frames between two manually labeled keyframes by estimating its position in each intermediate frame. It lets annotators label sparse keyframes and have the rest generated automatically.
In this article
Building visual or physical AI?
Let's talk.
What is video annotation interpolation?
Interpolation is the workhorse that makes video annotation affordable. Instead of drawing a box on all 1,800 frames of a clip, an annotator labels the object on a handful of keyframes, the start, the end, and a few turning points, and the tool computes its position on every frame in between.
The simplest form is linear interpolation, where the box moves in a straight line at a steady rate between keyframes, and the annotator adds keyframes wherever the motion is not linear. The result is full per-frame labels from a fraction of the manual work.
Key takeaways
- Interpolation generates labels on intermediate frames from manually labeled keyframes.
- It turns labeling thousands of frames into labeling a few, with the tool filling the gaps.
- Accuracy depends on keyframe placement, so you add keyframes wherever motion changes.
What interpolation provides
| Step | What happens |
|---|---|
| Keyframes | The annotator labels an object on selected frames |
| Interpolation | The tool estimates the object's box or mask on the frames between them, usually linearly |
| Correction | The annotator adds keyframes where the interpolated path drifts from the object |
| Interpolation vs propagation | Interpolation is geometric estimation between human labels, propagation (SAM 2) predicts frames with a model |
How it works
You set keyframes and the labels fill in between, then scrub the timeline and fix where they drift.
Why it matters
Interpolation is the difference between video annotation being feasible and being prohibitively expensive. Its quality is entirely a function of where keyframes go, and the common mistake is spacing them evenly. Motion is not uniform, an object sits still, then darts, so evenly spaced keyframes over-label the still parts and under-label the fast ones, producing boxes that lag or overshoot exactly during the moments that matter. Smart keyframing follows the motion, dense where it changes, sparse where it does not.
Frequently asked questions
What is interpolation in video annotation?
Generating an object's labels on frames between keyframes by estimating its position, so you only hand-label keyframes.
What is the difference between interpolation and propagation?
Interpolation estimates geometrically between human-labeled keyframes. Propagation uses a model to predict frames, like SAM 2.
How many keyframes do you need?
As many as the motion requires, more where the object's path changes, fewer where it moves steadily.
Related terms
Director of Growth
Jesse Mostipak is the Director of Growth at Voxel51, where the work is helping humans find and trust what the brand knows, and teaching the Google knowledge graph and the LLMs answering on their behalf to do the same. That question, how knowledge gets built inside a system, is one Jesse has been chasing for years. Earlier versions of it ran through a New York City high school science classroom, data science and machine learning, and developer relations at Kaggle, Posit (formerly RStudio), and Baseten. The answer doesn't change much depending on whether the learner is a teenager, a software engineer, or a knowledge graph. Jesse holds a Master's in Education from CUNY Hunter College. LinkedIn
See all articles by Jesse Mostipak
Last updated November 3, 2026
Building visual or physical AI?
Let's talk.
© 2026 Voxel51 All Rights Reserved