Panoptic segmentation is a computer vision task and annotation type that labels every pixel in an image with both a class and, for countable objects, an instance identity, unifying semantic and instance segmentation into one complete, gap-free labeling of the scene.
Panoptic segmentation combines the two halves of segmentation into a single, complete labeling. It assigns every pixel a class, like semantic segmentation, and, for countable "things" like cars and people, also a unique instance ID, like instance segmentation, while uncountable "stuff" like road and sky gets a class but no instance.
The result leaves no pixel unlabeled and no object ambiguous: you know what every pixel is and which object it belongs to. "Panoptic" means everything visible, and that is the goal, one coherent map of the whole scene.
Key takeaways
Panoptic segmentation gives every pixel a class, and every countable object an instance ID.
It unifies semantic segmentation (stuff) and instance segmentation (things) into one complete map.
No pixel is left unlabeled and no object is ambiguous, so it is full scene coverage.
What panoptic segmentation provides
How panoptic segmentation treats "things" versus "stuff".
How panoptic segmentation treats "things" versus "stuff".
Category
Examples
Label
Things
Countable objects, car, person
A class and an instance ID
Stuff
Amorphous regions, road, sky, grass
A class, no instance
How it works
Annotators or models produce a complete per-pixel map where things are instance-separated and stuff is class-labeled. In FiftyOne, segmentation labels render as overlays and can be compared to predictions, so you can evaluate panoptic outputs and find where things are merged or stuff is misclassified.
Why it matters
Panoptic segmentation is the most complete scene description in 2D, which is why it suits robotics and driving, where a model needs both "what is everywhere" and "which object is which." It is scored with panoptic quality (PQ), which penalizes both classification and instance errors, so it is unforgiving in a useful way, because a model can ace semantic IoU and still post a poor PQ when it merges adjacent instances. It is also the most expensive segmentation to annotate, because it demands complete coverage and instance separation at once, so it is reserved for tasks that genuinely need both.
Frequently asked questions
What is the difference between panoptic and semantic segmentation?
Semantic labels every pixel by class. Panoptic also separates countable objects into instances.
What are "things" and "stuff" in panoptic segmentation?
Things are countable objects with instance IDs. Stuff is amorphous regions with only a class.
How is panoptic segmentation evaluated?
With panoptic quality (PQ), which accounts for both class and instance accuracy.