Event annotation, also called action or temporal annotation, is the labeling of what happens in time-based data and when, marking the start and end of actions, activities, or events in video or sensor streams. It captures behavior over time rather than objects in a frame.
Event annotation labels the temporal dimension: not what objects are present, but what is happening and over what span. An annotator marks a segment of a video or sensor stream, "person falls," "vehicle turns left," "machine starts," with a start time, an end time, and a label, sometimes tied to the objects involved.
It is the basis for action recognition, activity detection, and temporal localization, where the model must answer "when did X occur," not just "is X here." Events can be instantaneous, a moment, or extended, an interval, and the same clip can carry many overlapping events.
Key takeaways
Event annotation labels actions and events in time, with start and end boundaries.
It captures behavior over time, distinct from object detection (per frame) and tracking (identity over time).
It powers action recognition, activity detection, and temporal localization.
What event annotation provides
Types of event annotation.
Types of event annotation.
Type
What it is
Temporal segments
An action with a start and end (interval)
Point events
A single moment, a ball is kicked
Object-linked events
Actions tied to the agents performing them
Overlapping events
Multiple actions co-occurring in the same span
How it works
Annotators scrub the timeline and mark the boundaries of each action or event, assigning a label and optionally linking the objects involved. In FiftyOne, temporal labels can be stored on video and reviewed along the timeline, and compared to predicted events to find where boundaries or classes disagree.
Why it matters
Event annotation is how AI learns behavior, the layer above perception, essential for safety, sports, surveillance, and any system that must recognize what is going on. Like all temporal labeling, its dominant error source is boundary ambiguity, deciding the exact frame an action begins and ends, because actions blur at their edges, when exactly does "reaching" become "grabbing." Annotators disagree far more on boundaries than on the action's identity, so event datasets need explicit timing conventions, and models are often better evaluated with tolerance windows around boundaries than with exact-frame matching.
Frequently asked questions
What is event annotation?
Labeling actions and events in time-based data with start and end boundaries, the basis of action recognition.
How is it different from object tracking?
Tracking follows an object's identity across frames. Event annotation labels what is happening and when.
Why are event boundaries hard to annotate?
Actions blur at their edges, so annotators disagree on the exact start and end frames more than on the action itself.