NVIDIA researchers' VFlash decodes robot trajectories in one pass, up to 12 times faster on edge GPUs

A team from NVIDIA's Learning and Perception Research group, working with Johns Hopkins University, Hong Kong Polytechnic University, Purdue University and the University of Hong Kong, has published VFlash, a decoding method meant to make vision-language models fast enough to plan robot motion on embedded hardware. The paper, titled "VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding", was released this week alongside a project page and code in NVIDIA's Eagle repository.

How it works

Vision-language models used in robotics and driving often describe a plan as a sequence of numbers, such as waypoints, bounding boxes or depth values, and generate it one token at a time. The authors measured that this autoregressive step takes 75% to 87% of model-stage latency on a Jetson Thor module. VFlash exploits the fact that these outputs have a known type and length: a tool call declares how many coordinates are needed, and a lightweight drafter of five non-causal transformer layers fills them all in a single pass, reading context that the frozen main model has encoded once. Unlike classic speculative decoding, the draft does not have to match the main model token by token; free-text sub-tasks are still drafted and then verified.

Reported results

On Jetson AGX Thor and DGX Spark, the team reports robot-planning decoding roughly 12 times faster, and up to 42 to 49 times faster for some driving workloads. On Thor, a full manipulation pipeline dropped from 3,552.7 milliseconds with standard decoding to 703.3 milliseconds, about 5.1 times quicker. The authors also report lower trajectory error, with mean 3D displacement error down 42.6% on average across three robot datasets and up to 61% in some settings.

In simulated manipulation on the MolmoSpaces benchmark, plans from VFlash fed a world action model built on NVIDIA's Cosmos-Predict2.5. Average success reached 70.2%, against 62.6% for the baseline policy and 61.0% when the plans came from ordinary decoding. On NAVSIM driving, a reinforcement-learning variant scored on par with the strongest systems listed.

Caveats

All figures come from the authors' own experiments and have not been independently reproduced. The results summarised on the project page rely on simulation benchmarks and recorded driving datasets rather than deployments on physical robots, and latency was measured at batch size one with custom TensorRT runtimes. The page links code but does not say whether trained weights are available. Interesting Engineering, which reported on the work on November 5, highlighted the speed and error figures.