Semantic segmentation is an annotation and prediction task that assigns a class label to every pixel in an image, grouping all pixels of the same category together without distinguishing individual objects. Every pixel gets a class, but two cars in one mask are not told apart.
Semantic segmentation labels an image pixel by pixel, so each pixel is assigned to a class like road, sky, car, or person. The output is a dense map the same size as the image where every location carries a category. It answers "what is at each pixel" but not "how many objects," so all the "car" pixels share one label whether there is one car or ten. That makes it ideal for scenes where coverage matters more than counting: drivable surface, land cover, tissue regions.
Key takeaways
Every pixel gets exactly one class label.
It does not separate individual objects, that is instance segmentation.
Output is a dense per-pixel class map, scored with IoU per class, often averaged as mean IoU.
How it works, and how FiftyOne fits
Annotators produce a per-pixel mask, increasingly with click-to-segment assistance, then review the boundaries. In FiftyOne, semantic masks render as colored overlays, and you can evaluate predicted maps against ground truth per class to see which categories the model handles well and which it confuses.
Semantic vs instance vs panoptic segmentation
Definitions for different types of segmentation
Definitions for different types of segmentation
Task
What it labels
Separates individual objects?
Semantic segmentation
Every pixel by class
No
Instance segmentation
Countable objects, one mask each
Yes
Panoptic segmentation
Every pixel by class, plus instance IDs
Yes for objects, background gets a class only
Where it's used
Autonomous driving: drivable area, lane surface, sidewalk.
Medical imaging: organ or lesion regions.
Geospatial and agriculture: land cover, crop versus soil.
Why it matters
Semantic segmentation is the right tool when you care about regions, not instances. Per-class IoU hides an imbalance problem. Large regions like road or sky dominate the average, while rare safety-critical classes, a pedestrian, a small lesion, can score terribly without moving mean IoU much. That is why teams report per-class IoU rather than just the mean, and curate hard examples of the rare classes instead of labeling more easy scenes.
Frequently asked questions
What is the difference between semantic and instance segmentation?
Semantic labels every pixel by class, instance also separates individual objects.
How is semantic segmentation scored?
With IoU per class, often averaged as mean IoU.
How is it different from object detection?
Detection draws boxes around objects, semantic segmentation labels every pixel.