Panoptic segmentation
Panoptic segmentation is a computer vision task and annotation type that labels every pixel in an image with both a class and, for countable objects, an instance identity, unifying semantic and instance segmentation into one complete, gap-free labeling of the scene.
In this article
Building visual or physical AI?
Let's talk.
What is panoptic segmentation?
Panoptic segmentation combines the two halves of segmentation into a single, complete labeling. It assigns every pixel a class, like semantic segmentation, and, for countable "things" like cars and people, also a unique instance ID, like instance segmentation, while uncountable "stuff" like road and sky gets a class but no instance.
The result leaves no pixel unlabeled and no object ambiguous: you know what every pixel is and which object it belongs to. "Panoptic" means everything visible, and that is the goal, one coherent map of the whole scene.
Key takeaways
- Panoptic segmentation gives every pixel a class, and every countable object an instance ID.
- It unifies semantic segmentation (stuff) and instance segmentation (things) into one complete map.
- No pixel is left unlabeled and no object is ambiguous, so it is full scene coverage.
What panoptic segmentation provides
| Category | Examples | Label |
|---|---|---|
| Things | Countable objects, car, person | A class and an instance ID |
| Stuff | Amorphous regions, road, sky, grass | A class, no instance |
How it works
Annotators or models produce a complete per-pixel map where things are instance-separated and stuff is class-labeled.
Why it matters
Panoptic segmentation is the most complete scene description in 2D, which is why it suits robotics and driving, where a model needs both "what is everywhere" and "which object is which." It is scored with panoptic quality (PQ), which penalizes both classification and instance errors, so it is unforgiving in a useful way, because a model can ace semantic IoU and still post a poor PQ when it merges adjacent instances. It is also the most expensive segmentation to annotate, because it demands complete coverage and instance separation at once, so it is reserved for tasks that genuinely need both.
Frequently asked questions
What is the difference between panoptic and semantic segmentation?
Semantic labels every pixel by class. Panoptic also separates countable objects into instances.
What are "things" and "stuff" in panoptic segmentation?
Things are countable objects with instance IDs. Stuff is amorphous regions with only a class.
How is panoptic segmentation evaluated?
With panoptic quality (PQ), which accounts for both class and instance accuracy.
Related terms
Director of Growth
Jesse Mostipak is the Director of Growth at Voxel51, where the work is helping humans find and trust what the brand knows, and teaching the Google knowledge graph and the LLMs answering on their behalf to do the same. That question, how knowledge gets built inside a system, is one Jesse has been chasing for years. Earlier versions of it ran through a New York City high school science classroom, data science and machine learning, and developer relations at Kaggle, Posit (formerly RStudio), and Baseten. The answer doesn't change much depending on whether the learner is a teenager, a software engineer, or a knowledge graph. Jesse holds a Master's in Education from CUNY Hunter College. LinkedIn
See all articles by Jesse Mostipak
Last updated November 3, 2026
Building visual or physical AI?
Let's talk.
© 2026 Voxel51 All Rights Reserved