Relationship annotation is the labeling of how objects in a scene relate to one another, such as "person riding bicycle" or "cup on table," capturing interactions and spatial relations beyond individual objects. When these relations are linked across a whole image, the result is a scene graph.
Relationship annotation goes beyond labeling individual objects to label the connections between them. Each relationship is typically a triplet, subject, predicate, object, like (person, riding, horse) or (bottle, on, shelf), capturing actions, spatial relations, and interactions. Annotators first label the objects, then draw the links and assign each a relationship type.
When you connect all the objects and their relationships across an image, you get a scene graph: a structured, graph representation of the whole scene where nodes are objects, with attributes, and edges are relationships. It is what lets a model reason about a scene as a system, not just a bag of detections.
Key takeaways
Relationship annotation labels how objects relate, usually as subject-predicate-object triplets.
A scene graph is the full graph of objects and their relationships across an image.
It captures interactions and context that object and attribute labels alone miss.
What relationship annotation provides
What relationship annotation captures.
What relationship annotation captures.
Type
Examples
Spatial relations
On, under, next to, behind
Actions and interactions
Riding, holding, wearing
Part-whole and ownership
Has, part of, belongs to
The graph
Objects as nodes, relationships as edges, attributes on the nodes
How it works
Annotators label objects, then connect pairs and choose a predicate for each link, building up the graph. In FiftyOne, you can store objects with their attributes and the relationships between them, so scene-graph-style data and its predictions can be visualized and compared.
Why it matters
Relationship annotation is what moves a model from "what is in the image" to "what is happening," the basis of scene understanding, visual question answering, and richer retrieval. The labeling cost is combinatorial and the signal is sparse. With N objects there are on the order of N-squared possible pairs, but only a handful are meaningful relationships, so most of the annotation effort risks going to empty or trivial pairs. The discipline that makes it tractable is labeling only salient relationships, not exhaustively, which means relationship datasets are defined as much by what annotators are told to ignore as by what they label.
Frequently asked questions
What is relationship annotation?
Labeling how objects relate, as subject-predicate-object triplets like "person riding bike."
What is a scene graph?
A structured representation of an image as objects (nodes) connected by relationships (edges), with attributes.
Why is relationship annotation hard?
The number of possible object pairs grows quadratically while meaningful relationships are sparse, so deciding what to label matters.