Click-to-segment
Click-to-segment is an interactive annotation technique where a single click on an object produces a full segmentation mask, with a model inferring the object's boundary from that one prompt. It replaces manual pixel painting or polygon drawing with a click and a quick correction.
In this article
Building visual or physical AI?
Let's talk.
What is click-to-segment?
Click-to-segment is the interaction at the heart of modern model-assisted labeling. Instead of tracing an outline vertex by vertex or painting pixels, the annotator clicks on an object and a segmentation model, typically a SAM-style foundation model, returns a complete mask.
Extra clicks refine it: a positive click adds a missed region, a negative click removes spillover. The result is mask-quality segmentation at a fraction of the effort, which is why it has become the default way to create masks in most annotation tools.
Key takeaways
- One click yields a full mask, with extra clicks to refine.
- It is the interaction layer that turns segmentation models like SAM into a labeling tool.
- It collapses the cost of mask annotation, the most expensive label type, to clicks plus corrections.
What click-to-segment provides
| Element | What it does |
|---|---|
| Positive and negative clicks | Add or remove regions to shape the mask |
| Box or scribble prompts | Some tools accept a rough box or stroke instead of a point |
| Model behind it | A promptable segmentation model, SAM, SAM 2, or SAM 3, does the inference |
| Propagation | In video, the mask carries across frames, the SAM 2 and SAM 3 extension |
How it works
A click becomes a prompt to a segmentation model, whose mask is written in as a label the annotator can accept or refine.
Why it matters
Click-to-segment is the single biggest reason mask annotation stopped being prohibitively slow, because it changed segmentation from a per-pixel chore into a per-object click. The catch is the same automation bias that haunts all assisted labeling. A one-click mask looks finished, so annotators stop scrutinizing the boundary, and subtle errors, a sliver of background or a merged neighbor, slip through because the result feels authoritative. The speed is real, but quality still depends on reviewing the edges the model got almost, but not quite, right.
Frequently asked questions
What is click-to-segment?
An annotation technique where clicking an object produces a segmentation mask from a model, refined with more clicks.
What model powers click-to-segment?
Usually a SAM-family promptable segmentation model.
How is it different from polygon annotation?
Polygon annotation places vertices by hand. Click-to-segment generates the mask from a click and you correct it.
Related terms
Director of Growth
Jesse Mostipak is the Director of Growth at Voxel51, where the work is helping humans find and trust what the brand knows, and teaching the Google knowledge graph and the LLMs answering on their behalf to do the same. That question, how knowledge gets built inside a system, is one Jesse has been chasing for years. Earlier versions of it ran through a New York City high school science classroom, data science and machine learning, and developer relations at Kaggle, Posit (formerly RStudio), and Baseten. The answer doesn't change much depending on whether the learner is a teenager, a software engineer, or a knowledge graph. Jesse holds a Master's in Education from CUNY Hunter College. LinkedIn
See all articles by Jesse Mostipak
Last updated November 3, 2026
Building visual or physical AI?
Let's talk.
© 2026 Voxel51 All Rights Reserved