Physical AI is artificial intelligence that perceives, reasons about, and acts in the physical world through a body, such as a robot arm, a humanoid, an autonomous vehicle, or a drone, and that learns those skills from data recorded in the real world or simulated from it. This guide covers what physical AI is, the kinds of systems and models behind it, how physical AI data is collected and curated, and how teams in robotics, autonomous vehicles, defense, agriculture, and manufacturing turn that data into systems that work outside the lab.
What is physical AI?
Physical AI is AI that closes the loop between perception and action in the real world. A physical AI system senses its environment through cameras, LiDAR, radar, touch, and its own joint sensors, decides what to do, and then does it, by moving an arm, steering a vehicle, or flying a drone. Then it senses the result and decides again. Generative AI produces text, images, or code. Physical AI produces motion, and its mistakes have physical consequences.
Physical AI is a broad term that covers robots, vehicles, drones, and other machines. What these share is not a form factor but a problem: each one has to perceive a messy, changing world through imperfect sensors and act safely in it, and each one learns that from data recorded in the physical world.
Physical AI has three constraints that digital AI mostly doesn't. It runs in real time, so a perception model or policy has milliseconds to respond, usually on the robot or vehicle's own computer rather than in a data center. It acts on the world, so mistakes can break hardware or hurt people. And the world it acts in never stops changing, so a system that works in one warehouse, city, or season can fail in the next.
Physical AI has become the industry umbrella for robotics, autonomous systems, and the research field of embodied AI, especially for systems built on large learned models rather than hand-written rules.
The term took off in industry after NVIDIA began describing physical AI as the next wave of AI in 2024, along with a "three computers" model: one to train models, one to simulate and test them, and one on the robot or vehicle to run them. Three shifts made it practical. Foundation models trained on internet-scale images and text gave robots a head start on perception and language. Simulation and generative world models made it possible to produce training data and test scenarios without a physical robot. And shared open datasets and formats, from Open X-Embodiment to LeRobot, let teams pool data instead of starting from zero.
Physical AI is AI that perceives and acts in the physical world through robots, vehicles, drones, and other machines.
Modern physical AI systems are built on learned models: perception models, robot policies, vision-language-action (VLA) models, and world models.
Physical AI data is multimodal by default: synchronized camera, LiDAR, radar, inertial measurement unit (IMU), and robot state streams. In robotics it is often recorded as MCAP or stored as episodes in formats such as LeRobot.
Data, not model architecture, is the most common reason physical AI systems fail, and curating the right data is how teams improve them.
Physical AI tutorials and guides from the Voxel51 blog
These posts cover physical AI data from collection through curation and evaluation. Each section below links to more.
Physical AI used to be built from separate, hand-tuned modules: perception, then planning, then control. Learned models now do more of that work, and four families matter most.
Perception models turn sensor data into a structured view of the world: object detection and segmentation in images, 3D detection in point clouds, tracking over time, and depth estimation. They remain the backbone of autonomous vehicles, drones, and inspection systems, and they are what most annotation effort goes into.
Robot policies map observations directly to actions. Imitation learning policies such as Action Chunking with Transformers (ACT) and Diffusion Policy learn from demonstrations, often for a single task. Reinforcement learning improves a policy through trial and error, usually in simulation first.
Vision-language-action models extend vision-language models to output robot actions. A VLA takes camera images and an instruction such as "put the cup in the sink" and produces the motion to do it. Examples include Physical Intelligence's π0 family, NVIDIA Isaac GR00T, Google DeepMind's Gemini Robotics models, Hugging Face's SmolVLA, and OpenVLA, a widely used academic baseline. VLAs are pretrained on pooled data from many robots and then fine-tuned on a target robot, which is why the quality and diversity of that pooled data matter so much.
World models learn how the world changes over time, so a system can predict what will happen next, generate realistic scenarios for training and testing, or plan by imagining outcomes before acting. World foundation models are world models pretrained on large amounts of video and sensor data so they can be adapted to many physical AI tasks. NVIDIA Cosmos is one, and it is now used to generate synthetic driving and robot data. In the 2026 State of Visual and Physical AI survey, 92% of practitioners agreed that world models and spatial intelligence will break into mainstream awareness in the near future.
Every one of the 709 practitioners in the 2026 State of Visual and Physical AI survey reported underperforming models. The top five causes were all data problems: insufficient training data for specific scenarios (57%), data quality issues such as occlusion, lighting, and artifacts (48%), domain shift between training and real-world data (43%), annotation errors (32%), and class imbalance (31%). Model architecture ranked below all of them, at 19%.
Physical AI data is hard for reasons that don't apply to text or web images. It is expensive to collect, because a robot demonstration takes a person and a robot in real time, and a test drive takes a vehicle and a safety driver. It is multimodal, so every stream has to stay synchronized and calibrated. It is dominated by routine, because the moments that teach a model something new, a failed grasp, a near miss, a sensor glitch, are rare. And it never stops changing, because new sites, weather, objects, and hardware revisions keep shifting the distribution.
That is why the teams that do best spend more time on data, not less. In the same survey, 58% of teams that described their results as exceptional spent more than half their project time on data work, compared with 21% of struggling teams.
Physical AI data comes from four main sources, and most teams mix them.
Teleoperation. A person controls the robot, through a leader arm, a VR headset, a gamepad, or a phone, and the robot records its cameras, joint states, and commanded actions as an episode. This is how most robot learning data is collected, from small community datasets recorded on low-cost arms to large lab collections such as DROID.
Fleet and field logging. Robots, vehicles, and drones already in operation record continuously. Many fleets keep a rolling buffer and save a recording only when a trigger fires, such as a disengagement, a teleoperator takeover, a failed pick, or a low-confidence prediction, because storing everything is too expensive.
Simulation and synthetic data. Simulators and generative world models produce scenarios that are rare, dangerous, or expensive to record, such as a child running into the road or an unusual object on a conveyor. Neural reconstruction turns real recordings into editable 3D scenes, so one real drive can become many simulated variations. In the 2026 survey, 63% of practitioners agreed that synthetic data will become the primary source of training data.
The catch is the sim-to-real gap: behavior that works in simulation often fails on real hardware, and synthetic data inherits every flaw in the real data it was built from. On the sim-to-real gap covers how to diagnose it.
Human video. Egocentric video of people doing tasks, recorded from head-mounted cameras, is a growing source of pretraining data for manipulation, because it is far cheaper to collect than robot demonstrations.
Whatever the source, the result lands in a recording or dataset format. Robot Operating System 2 (ROS 2) has recorded to MCAP by default since the Iron release in 2023, and robot learning datasets are often stored as episodes in LeRobot format.
Recording is only the start. Every physical AI team runs the same loop on its data, whatever the robot or vehicle: look at it, find what matters, label it, train on it, and test the result in simulation and on hardware. The table maps each stage to the guide that covers it. The failures that evaluation exposes become the next targets for collection and curation, which is the data flywheel every physical AI team is trying to spin faster.
Stage
What happens
Guide
Record
Sensors write synchronized streams to MCAP or a dataset format
A physical AI system is evaluated by what it does, not just what it predicts. Robot policies are measured by success rate across trials in simulation and on real robots. Autonomous vehicles are measured against scenarios within their operational design domain, by metrics such as disengagements, and they are developed and validated under safety standards such as ISO 26262 for functional safety and ISO 21448 for the safety of the intended functionality.
Aggregate numbers hide the failures that matter. A policy with an 80% success rate can fail every time the object sits near the table's edge, so teams slice evaluation by scenario, condition, and embodiment and trace each failure back to the training data. Failure mining and scenario mining turn those findings into the next round of data collection and curation.
The data problem is the same across physical AI. What changes is the body, the sensors, and where the long tail lives.
Robotics
Robot learning is where physical AI is moving fastest. Manipulation policies and VLAs learn from teleoperated demonstrations, humanoids push the size of the action space, autonomous mobile robots share space with people, and warehouse robots run at a scale where every rare failure eventually happens. Because robot data is slow to collect, the gains come from curating the highest-value episodes and balancing mixes across tasks and embodiments. Berkshire Grey's robotics team uses Voxel51 to investigate issues in its warehouse picking data three times faster, across datasets of more than a million robot picks.
Autonomous driving has the most mature data pipelines of any physical AI field and some of its largest datasets. Fleets record cameras, LiDAR, radar, and vehicle signals across millions of miles, and nearly all of it is routine. Progress depends on finding the rare scenarios in that data, labeling them well, and turning the best of them into simulation. The Volkswagen Group, with Porsche and CARIAD, is building a physical AI data flywheel on Voxel51, NVIDIA, and Microsoft Azure that is expected to cut repeated physical test drives by up to 90%.
Defense systems fuse electro-optical, infrared, radar, and synthetic aperture radar data for perception and tracking, often on air-gapped networks where data can't leave the facility and outsourcing annotation isn't an option. The data lifecycle is the same, but every tool has to run on premises.
Agricultural physical AI runs on tractors, sprayers, harvesters, and drones that combine RGB and multispectral imagery with GPS. The same field looks different every week of the season and every region it's grown in, so domain shift is constant and datasets have to cover the full calendar.
Drones pair cameras with IMU and GPS, and sometimes thermal or LiDAR sensors, for inspection, mapping, delivery, and defense. Their data is hard in its own ways: altitude changes the scale of every object, motion blur and vibration degrade frames, and visual-inertial odometry drifts over a long flight. Flight controllers such as PX4 log telemetry in their own formats, and drones built on ROS 2 record to MCAP, which keeps imagery and inertial data on one timeline for replay.
Inspection systems look for defects that are rare by design, on production lines, utility infrastructure, and pipes. The challenge is extreme class imbalance and labels that have to be right, because a missed defect costs more than a false alarm.
Physical AI sits on a shared stack of open standards. ROS 2 is the robotics middleware widely used across research and industry, and it has recorded to MCAP by default since 2023. Robot learning datasets are distributed in LeRobot and RLDS formats, and Open X-Embodiment pools data from many robot types. NVIDIA's Isaac, Omniverse, and Cosmos platforms are widely used for simulation and synthetic data. Annotation standards such as ASAM OpenLABEL define how labels move between tools, and benchmarks such as nuScenes and LIBERO define how models are compared.
Physical AI is artificial intelligence that perceives, reasons about, and acts in the physical world through a body, such as a robot, an autonomous vehicle, or a drone, and learns to do so from real or simulated physical-world data. It senses the environment, decides what to do, acts, and senses the result, in a continuous loop.
Generative AI produces digital outputs such as text, images, and code. Physical AI produces actions in the real world, such as moving a robot arm or steering a car. Agentic AI sits in between: it takes actions, but in software, by calling tools and APIs. Physical AI and generative AI increasingly overlap: VLAs and world models borrow architectures from generative AI, but their outputs are motion and predictions of the physical world.
Physical AI and embodied AI describe the same idea from different angles. Embodied AI is the research term, emphasizing that intelligence comes from a body interacting with the world. Physical AI is the broader industry term, covering robots, vehicles, drones, and other machines built on learned models.
Examples of physical AI include robot arms that learn manipulation from demonstrations, humanoid robots, warehouse picking robots, autonomous mobile robots, surgical robots, self-driving cars and trucks, delivery and inspection drones, agricultural machines that spot weeds, and visual inspection systems on production lines.
Physical AI needs multimodal sensor data recorded in the real world or in simulation: camera images and video, LiDAR point clouds, radar, IMU readings, and robot or vehicle state, synchronized on one timeline. Robot learning also needs the actions taken and task descriptions. Coverage of rare scenarios matters more than raw volume.