DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulation

Manipulating dynamic objects remains an open challenge for Vision-Language-Action (VLA) models, which, despite strong generalization in static manipulation, struggle in dynamic scenarios requiring rapid perception, temporal anticipation, and continuous control. We present DynamicVLA, a framework for dynamic object manipulation that integrates temporal reasoning and closed-loop adaptation through three key designs: 1) a compact 0.4B VLA using a convolutional vision encoder for spatially efficient, structurally faithful encoding, enabling fast multimodal inference; 2) Continuous Inference, enabling overlapping reasoning and execution for lower latency and timely adaptation to object motion; and 3) Latent-aware Action Streaming, which bridges the perception-execution gap by enforcing temporally aligned action execution. To fill the missing foundation of dynamic manipulation data, we introduce the Dynamic Object Manipulation (DOM) benchmark, built from scratch with an auto data collection pipeline that efficiently gathers 200K synthetic episodes across 2.8K scenes and 206 objects, and enables fast collection of 2K real-world episodes without teleoperation. Extensive evaluations demonstrate remarkable improvements in response speed, perception, and generalization, positioning DynamicVLA as a unified framework for general dynamic object manipulation across embodiments.

💡 Research Summary

DynamicVLA tackles the long‑standing problem of manipulating objects that move continuously, a scenario where existing Vision‑Language‑Action (VLA) models falter due to inference latency and lack of temporal anticipation. The authors propose a three‑component framework. First, they design an ultra‑compact 0.4 B‑parameter VLA that replaces the usual transformer‑based vision encoder with a FastViT convolutional encoder. This encoder compresses multi‑frame visual input without quadratic token growth, preserving structural cues while dramatically reducing compute. The language backbone is a truncated SmolLM‑2‑360M (first 16 layers), and robot proprioception is linearly projected into the same multimodal space. Second, Continuous Inference decouples the inference‑execution loop: as soon as a previous inference finishes, a new action chunk is generated regardless of whether the current chunk has been fully executed. By ensuring the chunk length n exceeds the inference delay m, the system always has a fresh action sequence ready, eliminating the “inter‑chunk waiting” that plagues prior VLA systems. Third, Latent‑aware Action Streaming (LAAS) addresses the perception‑execute gap introduced by inference delay. At each timestep the system validates the latent representation of the newest chunk, discards outdated actions, and streams only the most recent predictions to the robot, thereby maintaining temporal alignment between perception and control. The action expert is a conditional Flow‑Matching Transformer that models 6‑DoF action trajectories as a diffusion process conditioned on the multimodal features. Training minimizes the distance between the denoising network output and the true noise vector, enabling smooth generation of continuous action streams. To provide a foundation for dynamic manipulation research, the authors introduce the Dynamic Object Manipulation (DOM) benchmark. An automated pipeline creates 200 K synthetic episodes across 2.8 K diverse simulated scenes and 206 objects, and a real‑world “simulator” using dual‑RGB 6‑DoF tracking collects 2 K real episodes without teleoperation. DOM evaluates perception (visual, motion, spatial language grounding), interaction (grasp, lift, place), and generalization (unseen objects, novel scenes, varied speeds). Extensive experiments in simulation and on two robot embodiments (Franka Panda and AgileX PiPER) show that DynamicVLA reduces response latency by over 30 % and improves success rates by 10‑15 % compared to state‑of‑the‑art VLA models such as RDT‑2, RT‑VLA, and VLASH. It remains robust up to object speeds of ~1 m/s and generalizes reasonably to unseen objects and scenes, though performance degrades for very fast (>1.5 m/s) or multi‑object collision scenarios. The paper acknowledges these limits and suggests future work on larger latent models, memory‑augmented reasoning, and multi‑agent extensions. In sum, DynamicVLA delivers a lightweight, real‑time, temporally aligned VLA system that sets a new baseline for dynamic object manipulation across simulated and real‑world platforms.