Image segmentation is the task of partitioning an image into regions by assigning a label to each pixel, rather than drawing a box around an object. It is the umbrella for semantic, instance, and panoptic segmentation, which differ in how they treat classes and objects.
Image segmentation delineates what is in an image at the pixel level. Instead of a coarse bounding box, it assigns a label to every pixel, producing precise masks that follow the exact shape of regions and objects. This fine-grained output is essential wherever the exact boundary matters, from medical imaging to scene understanding.
It is an umbrella term covering several variants that differ in what the labels mean.
Key takeaways
Segmentation labels every pixel, producing precise masks.
It is more fine-grained than bounding-box detection.
Semantic, instance, and panoptic variants differ in their labels.
What the main variants do
How the three main types of image segmentation differ.
How the three main types of image segmentation differ.
Type
What it labels
Semantic segmentation
Each pixel by class, without separating object instances
Instance segmentation
Each object instance separately, with its own mask
Panoptic segmentation
Every pixel, unifying class regions and distinct instances
How it works
A segmentation model outputs a class, and for instance and panoptic variants an instance identity, for each pixel, producing masks rather than boxes. Training requires pixel-accurate annotations, and quality is measured with overlap metrics like mean IoU and the Dice coefficient. The right variant depends on whether you need to separate individual objects, label regions, or both.
Why it matters
Precise masks are required wherever shape and exact extent matter, such as delineating a tumor, parsing a road scene, or measuring an object. Segmentation is a foundational visual AI capability, and understanding its variants clarifies how tasks like semantic, instance, and panoptic segmentation relate.
Frequently asked questions
How is segmentation different from object detection?
Detection localizes objects with boxes, while segmentation labels individual pixels, producing masks that follow exact shapes.
What are the main types of image segmentation?
Semantic segmentation labels pixels by class, instance segmentation separates individual objects, and panoptic segmentation combines both.