Vision-Language-Action model (VLA)
A Vision-Language-Action (VLA) model is a type of AI model that maps visual observations and a natural language instruction directly to robot actions. It extends the vision-language models used for image and text understanding by adding an action output, so a robot can turn a goal like "pick up the red block" into motor commands. VLAs have become a leading approach to building general-purpose robot control.
In this article
What is a Vision-Language-Action model (VLA)?Key takeawaysWhat a Vision-Language-Action model providesHow it worksBefore you fine-tune a VLA, read the episodesNotable VLA modelsVLA vs. VLM vs. world modelWhere VLAs are usedTraining data for VLA modelsWhy it mattersFrequently asked questionsRelated terms
In this article
What is a Vision-Language-Action model (VLA)?Key takeawaysWhat a Vision-Language-Action model providesHow it worksBefore you fine-tune a VLA, read the episodesNotable VLA modelsVLA vs. VLM vs. world modelWhere VLAs are usedTraining data for VLA modelsWhy it mattersFrequently asked questionsRelated terms
Building visual or physical AI?
Let's talk.
What is a Vision-Language-Action model (VLA)?
A Vision-Language-Action model, or VLA, is a model that takes in what a robot sees along with an instruction written in plain language, and produces the actions the robot should take. Depending on the robot, the output might be joint movements, an end-effector pose, or a gripper opening and closing.
The design builds on vision-language models (VLMs), which learn connections between images and text from internet-scale data. A VLA reuses that pretrained understanding and adds a way to emit actions, so a robot can apply what it knows about a "mug" or a "drawer" to a physical task. The term was popularized by Google DeepMind's RT-2 in July 2023, whose paper named this category of models, and open models such as OpenVLA later made it easier to build on.
Key takeaways
- A VLA maps camera images and a language instruction to robot actions, usually by extending a pretrained vision-language model.
- VLAs represent actions in two main ways: as discrete tokens generated like text, or as continuous values from a diffusion or flow matching action head.
- Some VLAs run as one model at a single rate, while dual-system designs couple a slow VLM with faster motor policies.
- Data is the bottleneck: a VLA can only be as good as the demonstration episodes it learns from.
What a Vision-Language-Action model provides
| Ingredient | Role in the model |
|---|---|
| Vision | Encodes what the robot currently sees from one or more cameras |
| Language | Interprets a natural language instruction or goal |
| Action | Produces executable commands such as joint angles, end-effector motion, or gripper state |
How it works
Most VLAs start from a pretrained VLM backbone that encodes the camera images and the instruction. RT-2 and OpenVLA then emit actions as tokens from the same language head the VLM uses for text. Others, such as π0 (pi-zero) from Physical Intelligence, add a separate action decoder, also called an action head or action expert, that outputs continuous commands.
VLAs are trained on triples of an instruction, an observation, and an action trajectory, usually from human demonstrations, and learn to reproduce the demonstrated actions, a form of imitation learning. Many VLAs predict a short sequence of future actions at once, a technique called action chunking, which smooths motion and limits errors that compound step by step.
How VLAs represent actions
VLAs represent actions as discrete tokens or as continuous values, and continuous action heads such as π0's usually use diffusion or flow matching to refine noise into a smooth trajectory.
| Approach | How it works | Example models | Tradeoff |
|---|---|---|---|
| Discrete action tokens | Each action dimension is split into bins, and the model outputs bin indices as tokens | RT-2, OpenVLA | Simple to train, but limited precision and control frequency |
| Continuous actions (diffusion or flow matching) | An action head turns noise into a continuous chunk of actions | π0, Octo, SmolVLA, GR00T N1 | Better for dexterous, high-frequency control and many degrees of freedom, with a more complex decoder |
The line between the two isn't fixed. OpenVLA-OFT (2025) kept OpenVLA's backbone but switched to continuous actions with parallel decoding. A middle path is FAST, used in π0-FAST, which compresses a chunk of actions with a discrete cosine transform before tokenizing it, so token-based models can learn dexterous, high-frequency skills.
Single-model vs. dual-system architectures
What separates single-model and dual-system VLAs is timing: one model running at a single rate, or two or more coupled models running at different rates.
It isn't one network versus two, since π0 and GR00T N1 both pair a VLM with an action expert. π0 runs both parts together at one rate. In a dual-system architecture, a slow VLM interprets the scene while a faster policy keeps acting on its latest output. Figure's Helix pairs a 7B-parameter VLM running at 7 to 9 Hz with an 80M-parameter policy running at 200 Hz, and NVIDIA's GR00T N1 pairs a 10 Hz VLM with a 120 Hz action module. Helix 02 (January 2026) adds a third, faster layer under those two systems, a whole-body controller running at 1 kHz.
A single rate is simpler to train, but a large model can only run so many times per second. Decoupling lets the motor layer react faster, which matters for humanoid robots with many joints. A related pattern pairs a VLA with a planner: Google DeepMind's Gemini Robotics shipped alongside an embodied reasoning (ER) model from the start, and since version 1.5 the ER model plans tasks and orchestrates the VLA.
Before you fine-tune a VLA, read the episodes
Fine-tuning a vision-language-action model on your own robot often starts with a few hundred demonstrations, and each one teaches the policy something, whether you meant it to or not. The problems that hurt most are quiet:
- a demonstration that ends in a failed grasp but stays in the training set
- an instruction that describes a different task than the one the operator performed
- two tasks with a thousand examples each and a third with forty
None of this shows up in a file listing or a loss curve.
Voxel51 opens each episode with its camera views on one timeline and its instruction, outcome, and other fields alongside. You can filter by outcome, lab, or number of grasps, group by task to see how balanced the mix really is, and ask the Voxel51 Agent to find behavior in plain language. The goal is a training set you've actually looked at before you spend GPU hours on it.
Robotics Droid
1,000 episodes
Real-world robot manipulation episodes from DROID, stored as MCAP with three synced cameras and fields for task prompt, success, and number of grasps.
Try this: Open the task_prompt filter. One task family, moving an object into or out of a container, covers 227 of the 1,000 episodes.
Explore DROID in Voxel51
Free with a work or .edu email.
Go deeper
Notable VLA models
Notable VLA model families include Google DeepMind's RT-2 and Gemini Robotics, OpenVLA, Physical Intelligence's π0 series, Figure's Helix, NVIDIA's GR00T N1 and Alpamayo, Hugging Face's SmolVLA, and Ai2's MolmoAct, each linked below to the developer's paper or announcement.
| Model family | Developer | First and latest release | Open weights | Action output | Architecture |
|---|---|---|---|---|---|
| RT-2 | Google DeepMind | Jul 2023 No later version | No | Discrete tokens | Single VLM, up to 55B |
| OpenVLA and OpenVLA-OFT | Stanford-led | Jun 2024 Latest: OpenVLA-OFT, Feb 2025 | Yes | Discrete tokens, continuous in OFT | 7B VLM on Llama 2 |
| Octo | UC Berkeley-led | Dec 2023 Latest: Octo 1.5, May 2024 | Yes | Diffusion head | Generalist robot policy, no VLM |
| TinyVLA | Midea Group-led | Sep 2024 No later version | Not confirmed | Diffusion head | Small VLM plus diffusion head |
| π0 family | Physical Intelligence | π0: Oct 2024 Latest: π0.7, Apr 2026 | π0 to π0.5: yes (openpi). Newer: Pi API only | Flow matching, FAST tokens in π0-FAST | VLM plus action expert |
| Helix | Figure AI | Feb 2025 Latest: Helix 2.5, Sep 2026 | No | Continuous | Slow VLM plus fast policy |
| GR00T N1 family | NVIDIA | Mar 2025 (N1) Latest: N1.7, Apr 2026 | Yes. N1 to N1.6 non-commercial, N1.7 commercial use allowed | Flow matching | VLM plus fast diffusion transformer |
| Gemini Robotics | Google DeepMind | Mar 2025 Latest: Gemini Robotics 2, Jul 2026 | No ( | Continuous values | VLA plus embodied reasoning model |
| SmolVLA | Hugging Face | Jun 2025 No later version | Yes | Flow matching | Small VLM plus action expert |
| Alpamayo | NVIDIA | Oct 2025 Latest: 2 Super, Aug 2026 | Yes | Driving trajectory plus reasoning text | VLM plus diffusion trajectory decoder |
| MolmoAct | Ai2 | Aug 2025 Latest: MolmoAct 2, May 2026 | Yes | Tokens, flow matching in MolmoAct 2 | VLM, action expert in v2 |
The field moves fast, and this table reflects releases as of November 2, 2026. Figure says Helix 2.5 was pretrained from scratch on its Index dataset rather than from a VLM, and hasn't published its architecture, so the Helix row describes Helix and Helix 02. Licenses: MIT for OpenVLA (its weights also fall under the Llama 2 Community License), Octo, and TinyVLA's code. Apache 2.0 for the openpi repository, SmolVLA, and MolmoAct 2. OpenMDW-1.1 for Alpamayo. NVIDIA Open Model License for GR00T N1.7.
VLA vs. VLM vs. world model
A VLM describes what it sees, a VLA decides what a robot should do, and a world model predicts what will happen next.
| Model type | Inputs | Outputs | What it's trained on | Typical use |
|---|---|---|---|---|
| VLM | Images and text | Text | Image and text pairs, mostly from the web | Captioning, visual question answering |
| VLA | Camera images, an instruction, and often joint state | Robot actions | A pretrained VLM, then robot demonstrations | Following instructions on a robot |
| World model | Observations and, often, an action | Predicted future observations | Video and interaction data | Simulation, planning, and policy evaluation |
The three are often combined. A world model can generate synthetic trajectories to post-train a VLA, as in NVIDIA's GR00T-Dreams workflow, or serve as a learned simulator for evaluating policies, as 1X has described.
ACT (Action Chunking with Transformers) isn't a VLA. It's an imitation learning policy trained from scratch per task on camera images and joint positions, with no language input or VLM pretraining. Its action chunking idea now appears in VLAs such as π0 and SmolVLA.
Where VLAs are used
VLAs suit robots that must follow open-ended instructions in changing settings.
- Tabletop and mobile manipulation. Most VLA research targets arms that pick, place, and rearrange objects. Physical Intelligence tested π0.5 on tasks such as cleaning a kitchen or bedroom in homes it had never seen.
- Humanoids. Humanoid robots need many joints coordinated at once. Figure's Helix and NVIDIA's GR00T N1 were both built for humanoids, and both use dual-system designs.
- Autonomous driving. Driving VLAs take camera video and output a planned path for the vehicle. NVIDIA's Alpamayo-R1, now called Alpamayo 1, outputs a trajectory with a written reasoning trace. XPeng's VLA 2.0, rolled out in China in March 2026, keeps the name but, by XPeng's description, drops the explicit language step.
Training data for VLA models
VLAs learn from episodes, recordings of a robot doing a task paired with its instruction. An episode usually contains:
- Camera views, often wrist and scene cameras
- Robot state, or proprioception, such as joint positions and gripper opening
- Actions sent to the motors at each timestep
- A task instruction, often written during task annotation
- An outcome, such as success or failure
Most of this data comes from teleoperation, using VR controllers or a leader-follower rig that the robot mirrors. Teams also add simulation and human video, as NVIDIA did for GR00T N1. For setup, latency, and labeling advice, see how to collect teleoperation data for robot learning.
A handful of public datasets anchor the field:
- Open X-Embodiment: more than 1 million real robot trajectories from 22 robot embodiments, pooled from 60 existing datasets
- DROID: 76,000 teleoperated trajectories, or 350 hours, recorded on Franka Panda arms in 564 scenes
- BridgeData V2: 60,096 trajectories on a WidowX 250 arm across 24 environments, each with a language instruction
- LIBERO: a simulation benchmark of 130 tasks in four suites, with human demonstrations for every task
- LeRobot community datasets: thousands of robotics datasets shared on the Hugging Face Hub
Two formats are common. RLDS, from Google Research, stores episodes as sequences of steps and is what Open X-Embodiment uses. LeRobotDataset, from Hugging Face's LeRobot, stores states and actions in Parquet and camera streams as MP4.
Curation matters because a VLA learns from every episode, mistakes included. Failed attempts, mismatched instructions, and lopsided task counts all shape the policy without stopping training. Reviewing episodes first catches these problems, and LeRobot support in Voxel51 lets teams search, compare, and curate episodes before the next training run. Mixing robots adds its own challenges, covered under cross-embodiment.
Why it matters
VLAs matter because they point toward general-purpose robots that can be directed with ordinary language instead of task-specific programming. By grounding web-scale semantic knowledge in physical action, they open a path to robots that generalize to new objects and requests rather than being hand-tuned for a single job. They also make robot data the key ingredient, since a VLA is only as capable as the demonstrations it learns from. That places a premium on collecting, unifying, and curating diverse, high-quality trajectories, especially across different robot bodies, which remains one of the field's central challenges.
For fine-tuning GPU floors, robot fit, and LIBERO scores across today's open models, see VLA models compared.
For the data checks to run before you fine-tune a model, see how to check robot demonstrations before fine-tuning a VLA.
Frequently asked questions
What does VLA stand for in AI?
In AI, VLA stands for vision-language-action, a type of model that turns camera images and a language instruction into robot actions, while outside AI the acronym often refers to the Very Large Array radio observatory in New Mexico.
How is a VLA different from a VLM?
A vision-language model (VLM) understands images and text but does not act. A VLA takes a VLM and adds an action output, so instead of just describing a scene or answering a question, it can produce the commands a robot needs to physically carry out a task.
What is VLA model architecture?
Most VLAs start from a pretrained VLM backbone that encodes the images and the instruction, then either emit actions as tokens from the VLM itself or add a separate action decoder. They also differ in whether one model runs at a single rate or coupled models run at different rates, both covered under How it works.
What are the most widely used VLA models?
Widely cited families include Google DeepMind's RT-2 and Gemini Robotics, OpenVLA, the π0 series, NVIDIA's GR00T N1 series, Figure's Helix, and SmolVLA. See the Notable VLA models table for details.
Are there open-source VLA models?
Yes. OpenVLA, Octo, SmolVLA, MolmoAct, NVIDIA's GR00T N1 and Alpamayo models, and Physical Intelligence's π0, π0-FAST, and π0.5 (through openpi) have open weights, and many are on Hugging Face. Terms differ: GR00T N1 to N1.6 are non-commercial, N1.7 allows commercial use, and Alpamayo uses OpenMDW-1.1. Physical Intelligence's newer models are available through its hosted Pi API rather than as open weights.
What is the difference between a VLA and a world model?
A VLA decides what a robot should do, while a world model predicts what will happen, such as the next video frames after an action. World models can generate training data for a VLA or test one in a learned simulator.
Is ACT a VLA?
No. ACT (Action Chunking with Transformers) is an imitation learning policy trained from scratch for each task, with no language instruction or pretrained VLM. Several VLAs borrow its action chunking idea.
How are VLAs trained?
Most VLAs start from a pretrained VLM and are then trained with imitation learning, also called behavior cloning, on demonstrations, usually gathered through teleoperation. Some newer models add a stage that learns from the robot's own experience and corrections, such as Physical Intelligence's π*0.6.
What do VLAs output?
The output is an action rather than text. Depending on the model and robot, this can be discrete action tokens, continuous joint or end-effector commands, or gripper state, and many models predict several future actions at once to keep motion smooth.
Are VLAs the same as a policy?
They are closely related. In robotics, the learned function that maps an observation to an action is called a policy, and a VLA is one way to build such a policy, distinguished by its use of vision, language, and web-scale pretraining.
Do VLAs work across different robots?
Getting a single VLA to work across different robot bodies, known as cross-embodiment, is an active research goal rather than a solved problem. Robots differ in sensors, joints, and action spaces, which is a major reason teams work to standardize data across embodiments.
How are VLA models used in autonomous driving?
In driving, a VLA takes camera video and outputs a planned trajectory, often with a written explanation, as NVIDIA's Alpamayo models do. Naming varies, though: XPeng's VLA 2.0 maps vision directly to action without an explicit language step.
Related terms
Vision-Language Model (VLM), Policy, Action head / action decoder, Imitation learning / behavior cloning, Open X-Embodiment, LeRobot, World model, Dual-system architecture, Diffusion Policy, Action chunking, Action Chunking with Transformers (ACT), Cross-embodiment, Embodiment, Teleoperation, SmolVLA, Episode, Task annotation, Policy evaluation.
Director of Growth
Jesse Mostipak is the Director of Growth at Voxel51, where the work is helping humans find and trust what the brand knows, and teaching the Google knowledge graph and the LLMs answering on their behalf to do the same. That question, how knowledge gets built inside a system, is one Jesse has been chasing for years. Earlier versions of it ran through a New York City high school science classroom, data science and machine learning, and developer relations at Kaggle, Posit (formerly RStudio), and Baseten. The answer doesn't change much depending on whether the learner is a teenager, a software engineer, or a knowledge graph. Jesse holds a Master's in Education from CUNY Hunter College. LinkedIn
See all articles by Jesse Mostipak
Last updated November 12, 2026
Building visual or physical AI?
Let's talk.
© 2026 Voxel51 All Rights Reserved