Vision-Language-Action model (VLA)

A Vision-Language-Action model, or VLA, is a single neural network that takes in what a robot sees along with an instruction written in plain language, and produces the actions the robot should take. In other words, it treats robot control as one unified problem: given these camera images and this request, what should the motors do next. The output might be a sequence of joint movements, an end-effector pose, or a gripper opening and closing, depending on the robot.

The design builds on vision-language models, which already learn rich connections between images and text from large amounts of internet data. A VLA reuses that pretrained understanding and adds a way to emit actions, which is often described as a VLA being a vision-language model plus an action output. This lets a robot inherit broad semantic knowledge, such as what a "mug" or "drawer" is, and apply it to physical tasks. The term was popularized by the RT-2 work in 2023, and open efforts such as OpenVLA later made the approach more widely accessible to researchers.

A VLA passes camera images and an instruction through a shared backbone that fuses the two into a common representation, then converts that representation into actions. Some models express actions as discrete tokens, generating them the way a language model generates words, while others attach an action head that outputs continuous values, sometimes using a diffusion process or a small transformer. Many VLAs predict a short sequence of future actions at once rather than one step at a time, which tends to make motion smoother and reduces the error that can accumulate when each step is predicted in isolation. Training usually relies on imitation learning, where the model learns to reproduce expert demonstrations, often collected by having a human teleoperate the robot.

VLAs are exciting because they point toward general-purpose robots that can be directed with ordinary language instead of task-specific programming. By grounding web-scale semantic knowledge in physical action, they open a path to robots that generalize to new objects and requests rather than being hand-tuned for a single job. They also reframe robot data as the key ingredient, since a VLA is only as capable as the demonstrations it learns from. That places a premium on collecting, unifying, and curating diverse, high-quality trajectories, especially across different robot bodies, which remains one of the field's central challenges.

A vision-language model (VLM) understands images and text but does not act. A VLA takes a VLM and adds an action output, so instead of just describing a scene or answering a question, it can produce the commands a robot needs to physically carry out a task.

Most VLAs are trained with imitation learning, also called behavior cloning, where the model learns to mimic expert demonstrations. Those demonstrations are frequently gathered through teleoperation, with a human piloting the robot to perform the task correctly.

The output is an action rather than text. Depending on the model and robot, this can be discrete action tokens, continuous joint or end-effector commands, or gripper state, and many models predict several future actions at once to keep motion smooth.

They are closely related. In robotics, the learned function that maps an observation to an action is called a policy, and a VLA is one way to build such a policy, distinguished by its use of vision, language, and web-scale pretraining.

Getting a single VLA to work across different robot bodies, known as cross-embodiment, is an active research goal rather than a solved problem. It is difficult because different robots have different sensors, joints, and action spaces, which is a major reason teams work to standardize data across embodiments.

Vision-Language Model (VLM), Policy, Action head / action decoder, Imitation learning / behavior cloning, Open X-Embodiment.