Image captioning

Image captioning produces a written description of an image, such as "a dog catching a frisbee on a beach." It requires both recognizing what is in the scene and expressing it in fluent, relevant language, which is why it sits at the intersection of visual understanding and text generation.

Modern captioning is handled by vision-language models that process the image and generate the sentence end to end.

A vision-language model encodes the image into a representation the language component can condition on, then generates a caption word by word. These models are typically pretrained on large collections of image-text pairs, which teaches them to associate visual content with language. The same backbone often supports related tasks like visual question answering.

Captioning makes visual content searchable, accessible, and describable at scale, from alt text for accessibility to indexing large media libraries. It also serves as a building block for multimodal understanding, and captions can enrich datasets with text that powers semantic search over images.

Classification assigns a label from a fixed set, while captioning generates free-form natural language describing the whole scene.

Vision-language models, which combine a visual encoder with a language generator trained on large image-text datasets.

Vision-Language Model (VLM), Visual question answering (VQA), Embedding and semantic search, Multimodal data.