What is open-vocabulary detection?
Open-vocabulary detection breaks the usual limit that a detector can only find the classes it was trained on. Instead, you specify the categories you want at inference time, often as free-form text, and the model localizes them using the broad visual-language knowledge it learned during pretraining. Asking it to find a "forklift" works even if forklift was never a training label.
It brings the flexibility of zero-shot recognition to the localization task of detection.
Key takeaways
- It detects categories specified at inference, not fixed at training.
- Targets are often given as free-form text prompts.
- It relies on vision-language pretraining for open-ended recognition.
How it works
The approach couples detection with a vision-language model that aligns images and text in a shared embedding space, typically learned through contrastive training. Region proposals or predicted boxes are matched against the text embeddings of the requested categories, so the class set is defined by the prompt rather than a fixed classifier. Prompt wording noticeably affects results, as it does in zero-shot labeling.
Why it matters
Fixed-class detectors require retraining for every new category, which is slow and data-hungry, while open-vocabulary detection adapts to new concepts on the fly. That makes it powerful for exploration, rapid prototyping, and bootstrapping labels, and it is a growing part of modern visual AI.
Frequently asked questions
How is open-vocabulary detection different from standard detection?
Standard detection is limited to a fixed set of trained classes, while open-vocabulary detection can localize new categories specified at inference, often by text.
Why does the prompt matter?
Because detection is driven by matching image regions to the text of the requested category, the exact wording influences what the model finds, much like in zero-shot labeling.
Related terms