Skip to main content

Advances in OpenVLA: On how I can legitimize my crazy ambitious line of research as a complete outsider.

paper-breakdownroboticsvlavision-language-actionfine-tuningopen-sourceLoRA

As always, let’s start with the Why

This post is a breakdown of OpenVLA: An Open-Source Vision-Language-Action Model in my own words. This is a big one because it could be potentially groundbreaking for a whole field. Or a whole host of fields. First, let’s get to the paper itself.

Why this paper exists

The traditional way to teach a robot a new task - say, picking up an eggplant and putting it in a bowl - is to train a policy from scratch for that specific task on that specific robot. You collect demonstrations (a human teleoperates the robot doing the task hundreds of times), then train a neural network on those demonstrations. The policy you get works for that task on that robot in that environment. New task? Start over. New robot? Start over. Different lighting? Might break. Different friction due to rust or obstacles or whatever, might also break. Different robot? definitely start over.

Meanwhile, in the LLM world you and I already know, something interesting is happening. Well, actually, it already happened a while ago, I’m just now learning about it myself as of about a few months ago: models pretrained on massive internet text turned out to be useful for tasks they’d never been explicitly trained for.

A model trained on internet text can write code, even though nobody labeled “this is a coding example” in the training data. It can translate between languages it saw relatively little of. It can do arithmetic despite never being given a math textbook. The knowledge is a side effect of having seen enough of the internet. GPT-3 can do translation even though nobody labeled translation examples in its training data - the knowledge was absorbed during pretraining.

In our course at Newline, Professor Zao describes it in admittedly a bit of a hokey-sounding way: imagine that inside each large language model there’s a universe in a black box we can’t see into, but when we interface with that internal universe by way of how we formulate our prompts, we can help the LLM park it’s focus on to a particular galaxy of knowledge within that universe. Whoa. Pretty far out.

In reality it’s just a helpful mental model to conceptualize what it means to effectively work with prompts in AI/ML engineering and how that interacts with an LLM’s existing knowledge from the massive training it received in these gargantuan behemoth data centers we all keep hearing about in the news.

Here are some other examples of pretrained models being useful for things they weren’t trained for: Vision: CLIP (by OpenAI, 2021) was trained to match images with text captions billions of image-caption pairs scraped from the internet. They never trained it on “is this a hot dog?” but it turns out it can classify images into categories it’s never seen, because it learned a deep correspondence between visual concepts and language.

SigLIP (Google, 2023) does the same thing but better.

DINOv2 (Meta, 2023) was trained purely on images with no text at all - just self-supervised learning - and it learned to understand spatial structure, object boundaries, depth cues. Nobody taught it those things explicitly.

Like I said, this has been happening for a while now. I’m only just now hearing about it. And so it should come as no surprise to anyone that even in 2024 when this paper was published, even back then, people started asking: can we do the same thing for robots? Instead of training from scratch every time, what if we took a model that already understands vision and language (a VLM, short for vision-language model, like the things that power image captioning and visual question answering) and fine-tuned it to also output robot actions?

That’s a VLA, a vision-language-action model. The “action” part is the new part here: instead of outputting text (“this is an eggplant”), it outputs motor commands (“move arm 2cm left, rotate gripper 5 degrees, close grip”). What used to be sequences programmed for the specific hardware back in the day of teenage me with a couple of motors, can now be condensed to actions.

And even in this paper as a citation, it is clear that Google already proved these VLAs work with RT-2-X - a 55-billion parameter VLA that set the state of the art back in 2023.

Problem is, like much of Google’s cutting-edge projects: RT-2-X is closed. You can’t download it, you can’t inspect the training recipe, you can’t fine-tune it for your own robot. And at 55B parameters, it’s enormous… well, was enormous for 2024. These days, open-source models at 70B parameters run locally on your own machine through tools like Ollama. This is what people mean when they say the AI field is moving at a break-neck speed.

OpenVLA

7B parameters (7x smaller than RT-2-X), trained on 970k real-world robot demonstrations, and it outperforms RT-2-X by 16.5% absolute success rate across 29 tasks. Everything is released - weights, code, training data pipeline, and fine-tuning scripts on HuggingFace.

This is where it gets really, really interesting. What a time to be alive! I know, you must be asking yourself: “Chiara, what are you saying? Are you saying robotics are going open source?” Not exactly. I sincerely doubt all the biggest players in robotics are going to go open source anytime soon.

And at best, I’m just the messenger here. A mere student, still very much an outsider. I’m not saying anything other than what it says in this paper. It essentially claims that the robotics field needs what the LLM field already has - open foundation models that anyone can build on. And as an AI/ML enthusiast, I wholeheartedly agree. What they literally say is:

“We argue that to develop a rich foundation for future research and development, robotics needs open-source, generalist VLAs that support effective fine-tuning and adaptation, akin to the existing ecosystem around open-source language models.”

They explicitly cite LLaMA, Mistral, and Gemma as the analogy they’re reaching for with OpenVLA. So here’s where you, dear reader are probably itching to tell me: “Oh, so they’re trying to be the next Ollama but for VLA’s.” Kind of. I wouldn’t exactly put it like that, but they are saying that “Google has a closed 55B model that works great and nobody else can use it - we’re building the open alternative.” which is essentially the same move as Meta releasing LLaMA against GPT-4. And we all know how that turned out.

Except here it came from established robotics researchers, not a publicly-traded MAGNA-tier company (aka FAANG/MAANG) in tech.

I gotta hand it to them, that’s absolutely awesome. Full stop. I’m a big fan. And this level of open sourcing is probably going to eventually pave the way for many other developers-turned-researchers like me, putting us in the position to make significant contributions despite being a near-total outsider to the field. I mean, unless of course, you count my 7th grade science fair project. Truly, what a time to be alive.

Ok, back to the paper.

Are these “actions” just sequences of commands?

Yes, you’ve got the right idea. But to make this a reality, the authors here also use a clever encoding and I think that’s more important to talk about. Or at least, it points to a clear pattern for a comparison I want to make.

In this paper each action is a 7-dimensional vector:

  • 3 position deltas: move arm Δx, Δy, Δz (how far to move in each spatial direction)
  • 3 rotation deltas: Δroll, Δpitch, Δyaw (how much to rotate the gripper)
  • 1 gripper command: open or close

That’s it.

The next key part here is that they discretize each of those 7 dimensions into 256 bins (like dividing a ruler into 256 tick marks), then treat each bin number as a token - the same kind of token a language model outputs when generating text. So predicting a robot action becomes the same computation unit as predicting the next word in a sentence. The Llama 2 backbone doesn’t need any architectural changes - it just outputs 7 tokens instead of words.

As you can see, it’s looking more and more like what we’ve already seen with other forms of embedding in LLMs that allow other big breakthroughs, and that’s the comparison I wanna make: Text-to-image models already proved you could encode images as tokens and generate them with a transformer. Text-to-audio did the same for sound. OpenVLA extends the pattern to robot movements.

A more specific example of this pattern: The original DALL-E took images, chopped them into patches, quantized each patch into a discrete codebook entry (again, a token), and then used a transformer to generate those tokens autoregressively - same as generating text, just the tokens meant “image patches” instead of “words.” OpenVLA does the exact same thing but the tokens mean “motor commands.”

So a single “action” is something like: move 2cm right, 1cm forward, 0.5cm down, rotate 3 degrees clockwise, keep gripper open.

VLAs vs VLMs

Ok so I keep using this VLA acronym, and you might be more familiar with its foundation, the VLM. Let me make the distinction clear because it matters for understanding why this paper is significant.

A VLM - a vision-language model - takes an image and a text prompt and produces text output. You’ve probably used one without even thinking about it. Upload a photo to ChatGPT and ask “what’s in this picture?” - that’s a VLM at work. Image in, text out.

A VLA - a vision-language-action model - takes that same image and text prompt and produces actions instead. Robot motor commands. Same input, fundamentally different output.

I put this chart together to line the two up side by side, the way the paper frames it:

VLMVLA
InputImage + text promptImage + language instruction
OutputText tokens (words)Action tokens (motor commands)
Example inputPhoto of kitchen + “What’s on the counter?”Photo of kitchen + “Put the eggplant in the bowl”
Example output”There’s an eggplant and a bowl”Δx=+2cm, Δy=+1cm, Δz=-0.5cm, Δroll=0, Δpitch=3°, Δyaw=0, grip=open
ArchitectureVision encoder → projector → LLMSame. Literally the same.
TrainingNext-token prediction on textNext-token prediction on discretized actions

See that last row? The architecture is identical. The only difference is what the output tokens represent and what data you fine-tune on. A VLM is trained on internet image-text pairs. A VLA is a VLM that was further fine-tuned on robot demonstration data where the “text” output has been replaced with discretized motor commands. That’s it.

This is why advances in VLMs - better vision encoders, better language models, better training recipes - automatically benefit VLAs too. They’re the same thing under the hood, just pointed at a different output space.

The Architecture

So how did they actually put OpenVLA together? Think of it as three stages that fire every time the robot needs to decide what to do next.

Stage 1: The Eyes (Visual Encoder). The camera image passes through two pretrained vision models running in parallel - DINOv2 and SigLIP. Why two? Because they see different things.

SigLIP is good at semantic understanding - it knows that the red thing is a tomato and the green thing is a cucumber, because it was trained on billions of image-caption pairs.

DINOv2 is good at spatial understanding - it knows where objects are, their boundaries, their shapes, their depth relationships - because it was trained purely on images with self-supervised learning.

The image passes through both encoders separately, and their outputs get glued together side by side, what the field calls channel-wise concatenation, but they’re really just stacked together without blending them. Think of how two transparencies can get laid over the same photo — one sheet of “what,” one sheet of “where,” both readable at once, still retaining their respective info.

The combined result gives the model both “what is this thing” and “where exactly is it” information. Together the visual encoder has about 600 million parameters.

Stage 2: The Bridge (MLP Projector). This is a small 2-layer neural network that translates the visual tokens into the format Llama 2 expects. The vision models and the language model speak different “languages” internally - different dimensionalities, different representations. The projector is just a translator between them. It’s tiny compared to the other two components and I’m honestly kind of charmed by how simple it is.

Stage 3: The Brain (Llama 2 7B). The same Llama 2 you already know from the LLM world. It receives visual tokens from the projector interleaved with text instruction tokens from the Llama tokenizer. The input is formatted as a prompt: "What should the robot do to {task}? A:" - and the model’s “answer” is 7 action tokens.

Pretty cool, right? Of course, those tokens don’t map to English words. Picture popping the 256 rarest keys off Llama’s 32,000-key keyboard and relabeling them with robot-motion meanings — common keys untouched, so it still “types” English fine, but now those 256 mean action bin values instead of words.

So when Llama outputs a particular token, it doesn’t mean a word anymore - it means “action dimension 1, bin 128” which translates to a specific physical displacement. The cross-entropy loss during training is only computed on these action tokens, not the rest of the sequence.

Here’s what I found to be most surprising about all this: no new architecture was invented. They took an existing VLM called Prismatic-7B and changed what the output tokens mean. That’s why the whole thing is so scalable - all the existing infrastructure for training LLMs (PyTorch FSDP, FlashAttention, HuggingFace AutoModel) works out of the box with no modifications. It’s not a new machine. It’s the same machine, pointed at a new problem.

So, no streaming video?

Ok I have to be honest here. When I first started reading about this, I imagined something way more sophisticated. Multiple camera angles. Streaming video. The model watching everything in real time, tracking objects across frames, remembering what it just tried. Like how a human would remotely control a robot arm.

The reality is much simpler than that. Almost disappointingly simple, if I’m being honest, until you realize the simplicity is a whole achievement on its own.

OpenVLA uses one camera. A single 3rd-person RGB camera pointed at the robot and workspace - like someone watching over the robot’s shoulder. Not a gripper-mounted camera, not a depth sensor, not a multi-angle rig. One camera, one static image per inference step.

And here’s the part that really surprised me: the model has zero memory between steps. Each decision comes from a single photograph. It doesn’t know what it saw 200 milliseconds ago. It doesn’t know that it just tried to grab the eggplant and missed. Every single inference is completely independent - the model makes the best possible decision from one snapshot, then immediately forgets.

Apparently this is because the control loop runs at roughly 5-15 Hz depending on the robot setup - meaning 5-15 fresh photos and fresh decisions per second. From the outside it looks smooth and continuous, like the robot is “watching” the scene and reacting in real time. But internally it’s more like a flipbook. Each frame is its own isolated decision.

Why did the authors choose this? Because a VLM already knows how to process one image plus one text prompt. By keeping it to single images, they didn’t need a custom video encoder, a temporal memory module, or recurrent connections. Maximum simplicity means maximum reuse of existing infrastructure. And as I’ll get into in the limitations section - the authors are the first to admit that this is a tradeoff.

Does this mean the setup varies if you change the robot specs?

Yes and no.

The 7D action format (position + rotation + grip) is standardized across the training data - they curated a training subset from the community-built Open X-Embodiment dataset, keeping only single-arm manipulation tasks with at least one 3rd-person camera and single-arm end-effector control. But different robots have different physical dimensions, different joint limits, different speeds, and different control frequencies.

The WidowX runs at 5 Hz; the Franka-DROID setup runs at 15 Hz. Their workspaces are different sizes. Their grippers have different force profiles.

The model learns to handle this variation because it’s trained on 970k demonstrations from 70+ individual robot datasets spanning multiple robot types. The action discretization (those 256 bins per dimension) is calibrated to the data using percentiles — the same idea as “you scored in the 90th percentile,” just a marker for where a value sits once everything’s lined up smallest to largest. Here they take the 1st and 99th percentile of actions, so the bin boundaries hug the real range of movements instead of being hard-coded.

The way I understand it is this: when you fine-tune for a new robot, you’re essentially teaching the model: “on THIS robot, these token values correspond to THESE physical movements.” The 10-150 demonstrations you collect are enough for the model to recalibrate its action space to match your specific hardware.

The Fine-Tuning

This is where the paper gets really practical. Training OpenVLA from scratch took 64 A100 GPUs running for 14 days. That’s 21,500 A100-hours. Not exactly something you or I can do over a weekend. But the whole point of a foundation model is that you don’t train from scratch - you fine-tune.

The authors tested five different fine-tuning strategies on Franka Emika Panda robot arms doing 7 manipulation tasks, from pick-and-place to cleaning a table. Here’s how each one performed:

Full fine-tuning updates every single one of the 7.2 billion parameters. It achieved a 69.7% success rate, which is the ceiling - the best you can do if compute isn’t a constraint. But it requires 163.3 GB of GPU memory, meaning you need at least 2 A100 GPUs with FSDP (fully sharded data parallelism) to even fit it in memory.

Last layer only fine-tunes just the final transformer layer and the token embedding matrix - about 465 million parameters. It cratered to 30.3% success. The model needs deeper adaptation than just the output head to learn a new robot’s dynamics. This was clearly not enough.

Frozen vision freezes the visual encoder and fine-tunes everything else - 6.8 billion parameters. You’d think keeping the pretrained vision features intact would help (that’s actually the standard advice for VLM training). But it only hit 47.0%. The authors hypothesize that the pretrained vision backbone doesn’t capture the fine-grained spatial details needed for precise robot control. The robot needs to see differently than an image captioner does.

Sandwich fine-tuning unfreezes the vision encoder, the token embedding matrix, and the last transformer layer - about 914 million parameters. It reached 62.1% while using only 64 GB of memory. This is a smarter trade-off because it tunes the parts that matter most (the “bread” of the sandwich - the input and output layers) while leaving the middle of the LLM backbone frozen.

LoRA (Low-Rank Adaptation) applies small trainable adapter matrices to all linear layers across the entire model. At rank 32, it trains only 97.6 million parameters - 1.4% of the full model - and hits 68.2% success. At rank 64 (195 million parameters) it also hits 68.2%. In other words, LoRA at rank 32 matches full fine-tuning performance while training 73x fewer parameters and fitting on a single A100 GPU. The fine-tuning takes just 10-15 hours. That’s an 8x reduction in compute compared to full fine-tuning.

The LoRA result is the one that matters most for accessibility. It means someone with a single GPU can adapt this model to a new task in under a day. And if you’re wondering - yes, LoRA is already on my reading list. Stay Tuned to that post. When I get there, this will be a concrete example of why that technique is so important.

Looking at these results, I can’t help but wonder what the next generation of parameter-efficient techniques might unlock. Methods like QLoRA (which combines quantization with LoRA) could potentially push the hardware requirements even lower. The authors themselves don’t speculate much here, but the trajectory is clear: every new efficiency technique developed for LLMs can be tested on VLAs with minimal adaptation.

The Hardware

Let’s talk about what physical equipment was involved, because “hardware” here means two things: the robots and the GPUs.

Training hardware. The final OpenVLA model was trained on a cluster of 64 NVIDIA A100 GPUs for 14 days, with a batch size of 2048. That’s a total of 21,500 A100-hours. For reference, an A100 has 80 GB of HBM2e memory and up to 312 TFLOPS of bfloat16 compute. This is serious infrastructure - the kind of thing you find at Stanford or Google, not in a home office. But again, this is the pretraining cost. You don’t need to repeat it.

Inference hardware. During deployment, OpenVLA in bfloat16 precision requires 15 GB of GPU memory and runs at approximately 6 Hz on a single NVIDIA RTX 4090 (24 GB VRAM, Ada Lovelace architecture). On an A100 it hits about 4 Hz. On an older A5000 (24 GB, Ampere architecture) it’s about 3 Hz. The 1080Ti and 2080Ti (11 GB each) can’t fit the model alone - you’d need to shard across two cards, and even then you’re looking at sub-1 Hz inference, which is too slow for real-time control.

With 4-bit quantization the picture changes significantly:

PrecisionSuccess RateVRAM Required
bfloat1671.3 ± 4.8%16.8 GB
8-bit58.1 ± 5.1%10.2 GB
4-bit71.9 ± 4.7%7.0 GB

4-bit quantization matches full-precision performance while cutting memory by more than half to just 7 GB. That fits comfortably on a mid-range consumer GPU. The 8-bit result is actually worse - not because of accuracy loss, but because the added quantization operations slow down inference. Slower inference changes the system dynamics: the robot receives commands at a lower frequency, which affects how it moves. 4-bit avoids this because the reduced memory transfer actually speeds up throughput compared to 8-bit.

Robot hardware. The paper uses three different robot setups, and they span a wide range of accessibility:

The WidowX by Trossen Robotics is a 6-degree-of-freedom tabletop robot arm used in the BridgeData V2 evaluations. It has a parallel-jaw gripper, uses a single 3rd-person camera, and operates at a 5 Hz control frequency with a non-blocking controller. You can actually buy one of these - it’s a commercially available research platform in the $3,000-5,000 range. This is the most accessible option for someone entering the field.

The Franka Emika Panda is a 7-degree-of-freedom industrial research robot arm used for all the fine-tuning experiments. It’s table-mounted in the “Franka-Tabletop” setup (5 Hz control) and mounted on a movable standing desk in the “Franka-DROID” setup (15 Hz control). It has higher precision and payload capacity than the WidowX, which is why it’s the standard in academic robotics labs. It’s commercially available but significantly more expensive - in the $20,000-30,000+ range.

The Google Robot is a mobile manipulation robot from the RT-1 and RT-2 evaluations. It’s a full mobile platform with a robot arm mounted on a wheeled base. This is Google’s proprietary hardware - you cannot buy it, and the paper uses it only for out-of-the-box evaluation, not fine-tuning.

What’s keeping this from being truly Open Source

The code is MIT-licensed. The weights are on HuggingFace. The training data pipeline is documented. So what’s missing? You know, beyond the $20k robot arms…

The authors themselves lay most of this out in Section 6 of the paper. To be clear, these aren’t my own statements, they’re limitations the researchers explicitly acknowledge, so I’ll let them speak for themselves below.

Single-image observation only. OpenVLA sees one frame at a time with no memory of previous frames. Real-world robot tasks often require remembering what just happened - “I tried to grab it and it slipped, so I need to adjust my grip angle.” A human teleoperator would remember that. OpenVLA doesn’t. The authors note that exploring VLMs pretrained on interleaved image and text data could enable observation history in future VLA training.

“The current OpenVLA model has several limitations. First, it currently only supports single-image observations… Exploring the use of VLMs pretrained on interleaved image and text data may facilitate such flexible-input VLA fine-tuning.”

No proprioceptive input. The model doesn’t know the robot’s own joint angles, forces, or velocities. It controls the arm purely from what it sees, not what the arm feels. Think of moving your arm with your eyes open but no sense of touch and no awareness of where your joints are. In reality, real-world robot setups have a wide range of possible sensory inputs beyond a single camera, and the authors leave multi-modal sensory integration to future work.

“Expanding OpenVLA to support multiple image and proprioceptive inputs as well as observation history is an important avenue for future work.”

Single camera. No gripper-mounted camera, no depth sensor, no multi-angle rig. Just one RGB image from a 3rd-person viewpoint. For complex tasks where occlusion is a problem (the robot’s own arm blocks the view of the object), this is a real constraint.

“In reality, real-world robot setups are heterogeneous, with a wide range of possible sensory inputs.”

Inference speed is a bottleneck. At 6 Hz on an RTX 4090, OpenVLA is fast enough for tabletop pick-and-place at 5 Hz control frequency. But high-dexterity setups like ALOHA need 50 Hz for bimanual manipulation tasks. The authors suggest action chunking (predicting multiple future actions at once) and speculative decoding as potential remedies, but these remain unexplored for VLAs.

“improving the inference throughput of OpenVLA is critical to enable VLA control for high-frequency control setups such as ALOHA, which runs at 50Hz… Exploring the use of action chunking or alternative inference-time optimization techniques such as speculative decoding offer potential remedies.”

Sub-90% success rate. OpenVLA outperforms every prior generalist policy, but “outperforms” does not mean “reliable.” The best results in the paper top out in the 70-85% range depending on the task category. For a research demonstration, that’s impressive. For a farming robot that needs to pick thousands of strawberries a day, a 15-30% failure rate compounds fast.

“While OpenVLA outperforms prior generalist policies, it does not yet offer very high reliability on the tested tasks, typically achieving <90% success rate.”

The hardware isn’t open-source. This one is my own addition. Not anything the authors mentioned in their own paper, just an obvious limitation to this going fully open source ASAP.

The model is open. The robot arms are commercially available but not cheap. The WidowX at ~$3-5k is the most accessible, but a full research setup with sensors, mounting, compute, and the arm itself adds up. And the Google Robot used in some evaluations is entirely proprietary. The intelligence is open; the body is not.

None of these are dealbreakers. They’re the kind of limitations you’d expect from a first-generation open-source foundation model in a new domain. The LLM world had similar growing pains - early open models were slow, unreliable, and limited compared to their closed counterparts. What matters is the trajectory, and the trajectory here is pointed in the right direction.

Fine-tuning as the bridge to outsiders

The traditional path into robotics research looks like this: get a PhD in robotics, join a lab with expensive hardware and massive compute, spend years collecting data and training policies from scratch for one robot doing one task. The barrier to entry is enormous.

OpenVLA changes the equation. Here’s what you’d actually need to go from “outsider with AI/ML skills” to “running a robot manipulation policy”:

  1. A robot arm — a WidowX (~$3-5k) or even cheaper hobby-grade hardware
  2. A camera — one 3rd-person RGB camera
  3. 10-150 demonstrations — teleoperate the robot doing your task. For farming: pick a strawberry 50 times while the camera watches
  4. A single GPU — an A100 (or possibly a 4090 with quantization)
  5. 10-15 hours — LoRA fine-tuning on your demonstrations
  6. The OpenVLA codebase — free, MIT-licensed, on GitHub

That’s it. You don’t need Google’s 64-GPU cluster. You don’t need Google’s proprietary model. You don’t need Google’s proprietary robot. You don’t need years of robotics-specific training, your existing AI/ML skills (understanding fine-tuning, LoRA, tokenization, transformer architectures) transfer directly because a VLA is just a VLM pointed at a different output space.

The paper’s own fine-tuning results prove this works: OpenVLA fine-tuned on small datasets outperforms Diffusion Policy (a state-of-the-art approach trained from scratch) by 20.4% on diverse multi-instruction tasks. The pretrained knowledge from internet-scale vision-language data gives it a head start that from-scratch methods can’t match.

Is it perfect? No. Sub-90% success, single image, no proprioception, 6Hz max. But it’s a foundation — one that’s open, fine-tunable, and accessible to people who weren’t born in a robotics lab. For someone entering farming robotics research from an AI/ML background, this paper is essentially saying: the door is open, and you already have most of the keys.

Sources

The paper this post breaks down

Other papers referenced

Models, datasets & tooling

Background & explainers