Instance segmentation is an annotation and prediction task that outlines each individual object in an image at the pixel level, giving every object its own mask and identity. Unlike semantic segmentation, it tells two cars apart, combining the per-pixel detail of a mask with per-object separation.
Instance segmentation produces a separate pixel mask for every object, so the output is a set of individually identified objects, each with a class and its own boundary. It is essentially detection plus segmentation: the model both finds each object and traces its exact shape. Where semantic segmentation lumps all "person" pixels together, instance segmentation knows person 1 from person 2, which is what you need whenever counting or tracking individual objects matters.
Key takeaways
Each object gets its own pixel mask and identity.
It combines object separation, like detection, with exact shape, like a mask.
It covers "things," countable objects, not amorphous "stuff" like sky or road, which is semantic territory.
How it works, and how FiftyOne fits
Annotators outline or paint each object separately, often with click-to-segment assistance, then review for missed or merged objects. In FiftyOne, instance masks render per object with class and confidence, and evaluation matches predicted instances to ground truth by IoU, so you can surface merged, split, or missed objects.
Instance vs semantic vs panoptic segmentation
Comparisons between instance, semantic, and panoptic segmentation
Comparisons between instance, semantic, and panoptic segmentation
Task
What it labels
Object identity?
Instance segmentation
Countable objects, one mask each
Yes, each object is separate
Semantic segmentation
Every pixel by class
No
Panoptic segmentation
Every pixel by class, plus instance IDs
Yes for objects, background gets a class only
Where it's used
Robotics and manipulation: separating individual items to grasp.
Cell biology and medical imaging: counting and measuring individual structures.
Retail and inventory: counting individual products.
Why it matters
Instance segmentation is the choice when you need both shape and count. Its hardest failure mode is not classification but instance separation. Touching or overlapping objects get merged into one mask or split into two, and standard mask IoU can look fine while the instance count is wrong. That is why crowded scenes, a pile of cells, a shelf of identical items, are where instance models break, and where targeted annotation of dense, overlapping examples pays off most.
Frequently asked questions
What is the difference between instance and semantic segmentation?
Instance separates individual objects, semantic labels every pixel by class without identity.
Is instance segmentation the same as object detection?
No, detection gives boxes, instance segmentation gives a precise mask per object.
What is panoptic segmentation?
A combination that labels every pixel by class and gives countable objects instance IDs.