Semantic segmentation
Semantic segmentation is an annotation and prediction task that assigns a class label to every pixel in an image, grouping all pixels of the same category together without distinguishing individual objects. Every pixel gets a class, but two cars in one mask are not told apart.
In this article
Building visual or physical AI?
Let's talk.
What is semantic segmentation?
Semantic segmentation labels an image pixel by pixel, so each pixel is assigned to a class like road, sky, car, or person. The output is a dense map the same size as the image where every location carries a category. It answers "what is at each pixel" but not "how many objects," so all the "car" pixels share one label whether there is one car or ten. That makes it ideal for scenes where coverage matters more than counting: drivable surface, land cover, tissue regions.
Key takeaways
- Every pixel gets exactly one class label.
- It does not separate individual objects, that is instance segmentation.
- Output is a dense per-pixel class map, scored with IoU per class, often averaged as mean IoU.
How it works
Annotators produce a per-pixel mask, increasingly with click-to-segment assistance, then review the boundaries.
Semantic vs instance vs panoptic segmentation
| Task | What it labels | Separates individual objects? |
|---|---|---|
| Semantic segmentation | Every pixel by class | No |
| Instance segmentation | Countable objects, one mask each | Yes |
| Panoptic segmentation | Every pixel by class, plus instance IDs | Yes for objects, background gets a class only |
Where it's used
- Autonomous driving: drivable area, lane surface, sidewalk.
- Medical imaging: organ or lesion regions.
- Geospatial and agriculture: land cover, crop versus soil.
Why it matters
Semantic segmentation is the right tool when you care about regions, not instances. Per-class IoU hides an imbalance problem. Large regions like road or sky dominate the average, while rare safety-critical classes, a pedestrian, a small lesion, can score terribly without moving mean IoU much. That is why teams report per-class IoU rather than just the mean, and curate hard examples of the rare classes instead of labeling more easy scenes.
Frequently asked questions
What is the difference between semantic and instance segmentation?
Semantic labels every pixel by class, instance also separates individual objects.
How is semantic segmentation scored?
With IoU per class, often averaged as mean IoU.
How is it different from object detection?
Detection draws boxes around objects, semantic segmentation labels every pixel.
Related terms
Segmentation mask, Instance segmentation, Polygon annotation, Intersection over union (IoU), Annotation.
Learn more
See why pixel-level labels beat boxes when regions matter in Preventing Your Vision AI Models From Failing.
Director of Growth
Jesse Mostipak is the Director of Growth at Voxel51, where the work is helping humans find and trust what the brand knows, and teaching the Google knowledge graph and the LLMs answering on their behalf to do the same. That question, how knowledge gets built inside a system, is one Jesse has been chasing for years. Earlier versions of it ran through a New York City high school science classroom, data science and machine learning, and developer relations at Kaggle, Posit (formerly RStudio), and Baseten. The answer doesn't change much depending on whether the learner is a teenager, a software engineer, or a knowledge graph. Jesse holds a Master's in Education from CUNY Hunter College. LinkedIn
See all articles by Jesse Mostipak
Last updated November 3, 2026
Building visual or physical AI?
Let's talk.
© 2026 Voxel51 All Rights Reserved