A label schema is the defined structure of a labeling task, the set of classes, attributes, and label types a dataset uses, and how they are organized. It specifies exactly what can be labeled and how, so every annotation conforms to one consistent format.
A label schema is the blueprint for a dataset's labels. It enumerates the classes, car, pedestrian, sign, the label types each uses, box, mask, keypoint, and the attributes attached to them, a vehicle's color or a sign's state, along with rules like whether classes are mutually exclusive.
Designing the schema is the decision that shapes everything downstream: too coarse and the model cannot make distinctions you need, too fine and labeling cost explodes while annotators confuse near-identical classes. The schema is what the guidelines explain and what the exported labels conform to.
Key takeaways
A label schema defines the classes, label types, and attributes a dataset uses, and their structure.
It is the structural spec: guidelines explain how to apply it, the schema says what exists.
Schema design is a high-stakes early decision, because it sets what the model can ever learn to distinguish.
What a label schema provides
What a label schema specifies.
What a label schema specifies.
Element
What it defines
Classes
The categories that can be labeled
Label types
Box, mask, polygon, keypoint, or classification, per class
Attributes
Additional properties on a label, color, occlusion, state
The schema is defined up front, encoded in the annotation tool, and carried through to the exported dataset, balancing granularity against labeling cost, leaving room to add classes later, and aligning to the decisions the model must make. In FiftyOne, datasets store labels with their classes and attributes in a consistent structure, so you can validate that labels conform to the schema and spot classes that are confused or underused.
Why it matters
The label schema is one of the most consequential and least reversible annotation decisions, because changing it after labeling often means re-labeling. Schema granularity quietly caps model performance in both directions. Merge two classes that behave differently and the model can never separate them no matter how much data you add, split a class too finely and annotators cannot tell the subclasses apart, injecting noise that looks like a model problem. The right granularity is set by the decisions the system must make downstream, not by how the world could in principle be categorized.
Frequently asked questions
What is a label schema?
The defined set of classes, label types, and attributes a dataset uses, and how they are structured.
What is the difference between a label schema and an ontology?
A schema is the structural spec of labels. An ontology adds the relationships and hierarchy between them.
Why is schema design important?
It determines what distinctions the model can ever learn, and it is costly to change after labeling.