Vision-Language Model (VLM)

A vision-language model, or VLM, is a model that works with images and text at the same time, learning how the two relate. Given a photo, a VLM can describe what is in it, answer questions about it, or locate the thing a sentence refers to. It bridges the gap between seeing and reading, so that visual content can be discussed and reasoned about in natural language. This makes VLMs the backbone of a wide range of multimodal applications, from image captioning and visual question answering to grounding language in what a camera sees.

VLMs are trained on large amounts of paired image and text data, which teaches them the associations between visual patterns and the words people use to describe them. Well-known building blocks in this space learn to align images and text in a shared representation, and more recent VLMs combine a visual encoder with a language model so they can both perceive and converse. Because they carry broad knowledge learned from the internet, VLMs are increasingly used as the starting point for systems that need to act in the world, where their visual and semantic understanding gives those systems a head start.

A VLM typically pairs a visual encoder, which turns an image into a numerical representation, with a language model that can read and generate text, and a component that connects the two so information can flow between vision and language. Training on many image and text pairs teaches the model which visual patterns tend to go with which words and concepts. Some foundational approaches focus on aligning images and text so that matching pairs land close together in a shared space, which supports tasks like retrieval, while others integrate a full language model so the system can hold a conversation about what it sees. The common thread is a learned bridge between pixels and language.

VLMs matter because they let machines discuss and reason about the visual world in human terms, which unlocks a huge range of applications and serves as a foundation for others. For physical AI, they are especially important as the starting point for action models, since a system that already understands what a mug or a drawer is can learn to interact with those things far more efficiently than one starting from scratch. Anyone building multimodal systems benefits from understanding VLMs, because they increasingly sit at the core of how machines connect perception to language and, ultimately, to action.

A VLM understands images and text but does not act. A vision-language-action model, or VLA, extends a VLM by adding an action output, so it can produce the commands a robot needs rather than only describing or answering questions about a scene.

They are trained on large collections of paired images and text, which teach the model the associations between visual content and language. This lets them describe images, answer questions about them, and connect phrases to specific regions.

Because a VLM already carries broad visual and semantic knowledge learned from internet-scale data, it gives a robotic system a strong starting understanding of objects and scenes. Building action capabilities on top of that understanding is far more efficient than learning everything from robot data alone.

Vision-Language-Action model (VLA), Embedding and semantic search, Multimodal data, Policy, World model.