Getting started with VLAs

General VLA mechanism

A VLA is a model that predicts actions given a robot’s current state. In practice, most of the VLAs take text (instructions) and an image (robot camera sensor input) and output actions.

We’ll see in the upcoming sections what those building blocks look like, you’ll see that there’s a lot of resemblance with VLMs.

“move red pepper to tray”

→

Text encoder

→

Robot camera

→

Sensor (image) encoder

→

→

(Optional) Robot position

→

Actions decoder

In the diagram above, you can see how these components interact: the model acts as a central reasoning engine that takes in both a natural language instruction (e.g., “pick up the yellow mug”) and visual input from the robot’s camera sensors. These inputs are processed through an encoder (much like in a VLM) but instead of producing a textual description, the decoder outputs a sequence of low-level action tokens. These tokens, which can represent things like joint positions or end-effector deltas, are then translated into physical movements executed by the robot’s hardware.

You might have heard about imitation learning. Imitation learning models learn to reproduce behaviors from collected data, often within a limited set of tasks. While they can generalize within similar settings, they tend to struggle outside their training distribution. VLAs, on the other hand, are designed to improve generalization by combining vision-language pretraining with large-scale, diverse robot data. We’ll cover an example of imitation learning in the next section, which will help understand the difference between VLAs and imitation learning.

VLA building block: ACT

Take a look at the ACT architecture and compare it to the previous VLA diagram: do you notice any difference?

ACT is a pure imitation learning algorithm, it just learns how to do one task (e.g. pick a mug) with a succinct dataset. You can notice that it doesn’t take a text prompt as an input. In some robot platforms, there is one ACT trained for each one of the tasks the robot can execute.

Why do we need an ACT if the robot is just performing one action? In the real world, even if the action of picking a mug looks very repetitive, there can be some slight adjustments the robot needs to make based on its sensor inputs. Eg if the mug is slightly shifted side-ways, or if the mug’s color is different. ACT can help cover those cases, but won’t generalize to newer tasks. You’ll need one ACT per task (pick a mug, pick trash, point to bottle…).

How to encode action sequences?

In the paper, the action space is the absolute joint position for two robots, a 14-dimensional vector. This is because the robot used in this paper has 2 arms, each with 7 DoF. In the case of this paper, the outputs are the absolute position commands for the 7 degrees of freedom, but we’ll see in the next papers that the output can sometimes be position deltas.

How do we scale this to more tasks?

As you can notice, ACT uses all of the VLA building blocks. It also relies on the transformer architecture which we know can be scaled to Foundational Models that generalize with more parameters, and a lot of pretraining data. This is exactly what has been done in the literature and industry: use the ACT style of modeling, but scaling it to FM-sizes with massive pretraining datasets. Instead of training the transformers from scratch, VLAs building blocks are usually initialized with LLM or ViT Foundational Models.

In short, this is how we could summarize the relationship between ACTs and VLAs

VLAs = (vision-language foundation models) + (ACT-style action modeling) + (robot data scaling)

Autoregressive VLAs

In this section I’ll show two examples of one VLA model type. Those models rely on an image encoder, a text tokenizer, and an LLM decoder. The actions are decoded in the token space, with slight differences between different models.

RT-2

• company: Google Deepmind
• date released: July 2023
• size: 5B-55B

RT-2 (Robotics Transformer 2) is a vision-language-action model from Google DeepMind that translates natural language instructions and visual input directly into robot actions. It builds on large vision-language pretraining and learns to ground concepts like objects, goals, and instructions into physical behavior.

Instead of predicting continuous control signals directly, RT-2 outputs robot actions as discretized tokens (e.g., position, rotation, and gripper controls split into bins), allowing it to be trained like a language model using cross-entropy loss. This design makes it possible to jointly learn from web-scale vision-language data and robotic data, enabling more generalizable robot behavior.

Some specifics:

• RT-2 output form: terminate Δpos_x Δpos_y Δpos_z Δrot_x Δrot_y Δrot_z gripper_extension
• The continuous dimensions are discretized into 256 bins uniformly, 256 tokens are allocated to predict those numbers
• Embodiment: RT-2 is only trained on one type of robot
• Cross-entropy loss training objective (256 tokens)

In practice, RT-2 is the precursor of RT-X, and RT-X models are often used instead of RT-2. The RT-X platform builds upon the RT-2 foundation by extending the training recipe to a significantly broader range of datasets and robot types. Rather than specializing in one embodiment, RT-X leverages the Open X-Embodiment dataset, which aggregates data from dozens of distinct robots and thousands of tasks into a single, standardized format. In the released setup, this spans roughly 9+ embodiments, and everything is converted into a shared observation and action representation so the model can be trained jointly across them. The model itself outputs actions in this unified space, which keeps the architecture consistent regardless of the robot. However, at deployment time there is still a postprocessing step that maps these normalized outputs back into each robot’s calibrated control space (e.g., scaling, coordinate frames, and gripper conventions). So the main idea is: the dataset enables cross-robot training through standardization, but actual execution still depends on per-embodiment calibration at the action mapping level.

OpenVLA

• lab: Berkeley
• date released: June 2024
• size: 7B

OpenVLA builds on a very similar principle. It is more modular and open, based on open source blocks and open source datasets. Something worth noting in the architecture is that the image encoder is replaced by two FMs, a DINO encoder, but also a SigLIP encoder. It is fairly common to see SigLIP used as a vision encoder in robotics.

How does the OpenVLA action space differ from the RT action space? As you can see, both models are built on very similar embodiments (typically 7-DoF robotic arms with a gripper), but the way they represent and interface with actions is slightly different, and this is where calibration differences show up. RT-2 represents actions as discrete tokens in the language model vocabulary, essentially treating robot commands like words that get decoded into a fixed control space (e.g., end-effector deltas and gripper commands). This makes the system heavily dependent on a predefined action-token mapping, which needs careful calibration when moving to a new robot or control convention. OpenVLA also predicts tokens at the model level, but those tokens are not the final control signals - they are decoded into continuous robot actions through a learned action head trained across multiple embodiments. So both use a “de-tokenization” step, but in RT-style models this mapping is fixed by a predefined discretization scheme tied to a normalized action space, whereas in OpenVLA it is learned through a trained action decoding head, making it more flexible across different robot setups.

Flow-matching-based VLAs

Another family of VLA models does not rely on an autoregressive LLM to generate actions, but instead uses flow matching. If you’re unfamiliar with flow matching, you can think of it as a method that starts from random noise and gradually transforms it into a valid robot action or short trajectory, conditioned on the robot’s state, visual input, and instructions. I highly recommend this video from AI coffee break with Letitia which helps walking through the differences between diffusion and flow-matching

Why do we use flow matching in robotics? Robotic tasks often have many valid ways to be executed, so models need to learn a range of possible actions rather than a single trajectory. Generative approaches like flow matching address this by modeling a distribution over actions. Compared to diffusion, flow matching is much faster at inference (often under 10 steps), making it suitable for real-time use. It also works directly in continuous action space, avoiding the need for discretization like in many autoregressive approaches and enabling more precise control.

GR00T N1.5

• company: NVIDIA
• date released: June 2025
• size: 2.2B

GR00T N1.5 is an example of flow-matching VLA. You can see that it still has a VLM component, but the action decoding part of the model is pretty different from the previous models we’ve seen. The action generator starts from a noisy action, and often also takes the robot state as an input.

Some specificities about this model:

• It was pretrained with humanoid datasets, and finetuned with the OpenX embodiment datasets. The authors use an MLP per embodiment to project them to a shared embedding dimension as input to the DiT (green blocks on the diagram).
• The actions are predicted in chunks of 16 actions A_t = [a_t, a_t+1, …, a_t+H-1]
• The flow matching algorithm uses 4 denoising steps to produce a valid action for the robot.

Is flow-matching better than autoregressive VLAs?

Both are actively used in VLAs, and there’s no clear consensus yet on which one is best. Autoregressive models (LLM-style) are more established and easier to scale, but flow-matching approaches can be faster at inference since they can generate actions or trajectories in parallel instead of step-by-step. Some recent systems also combine the two: for example, π0.5 uses an LLM-like module to break down high-level goals into a sequence of low-level textual actions (kind of like a chain-of-thought), and then uses those to condition a flow-matching decoder that produces the actual robot actions.

Can I deploy a VLA zero-shot on my own robot?

We’ve seen in the previous papers that many VLAs are trained on large datasets spanning multiple embodiments (e.g., single-arm 7-DoF robots, dual-arm setups, or even humanoids). This helps with generalization, but it doesn’t fully solve cross-embodiment transfer. Most papers highlight strong zero-shot generalization to new tasks, objects, or environments, but not true zero-shot transfer to entirely new robots. In practice, when you change the embodiment, performance usually drops and you need some level of fine-tuning. For example, OpenVLA is trained on ~1M trajectories across many robot types and shows strong multi-task generalization, but still relies on efficient fine-tuning to adapt to new setups. Similarly, the RT-X / Open-X Embodiment work shows that training across many robots improves transfer, but models still require adaptation at deployment . There are also explicit studies showing that “out-of-the-box” VLAs fail when the embodiment changes (e.g., rigid arm → soft robot), and that targeted fine-tuning is needed to bridge that gap (see paper). On the architecture side, systems like GR00T try to handle embodiment differences with modular designs (e.g., separate high-level reasoning + low-level control), and in some cases lightweight adaptation layers (like small MLPs or control heads) can be enough to retarget the policy. But overall, the pattern is consistent: multi-embodiment training gives you a strong initialization, not true plug-and-play zero-shot transfer, so you should expect at least some fine-tuning when moving to a new robot.

Choose your model

We already covered some models in this blog post. Here is a summary card with more model information to get you started with a VLA.

Thanks for reading until here! I hope that this article helped you understand the structure of VLAs, what models exist out there, and which ones you could use in your next project!

References

• ACT paper
  https://arxiv.org/pdf/2304.13705
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
  https://arxiv.org/abs/2307.15818
• Open X-Embodiment: Robotic Learning Datasets and RT-X Models
  https://arxiv.org/abs/2310.08864
• OpenVLA: An Open-Source Vision-Language-Action Model
  https://arxiv.org/abs/2406.09246
• GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
  https://arxiv.org/abs/2503.14734
• π0.5: a Vision-Language-Action Model with Open-World Generalization
  https://arxiv.org/abs/2504.16054
• SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics (LeRobot ecosystem)
  https://arxiv.org/abs/2506.01844
• Bridging Embodiment Gaps: Deploying Vision-Language-Action Models on Soft Robots
  https://arxiv.org/pdf/2510.17369