Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flow

Controlling high-dimensional systems in biological and robotic applications is challenging due to expansive state-action spaces, where effective exploration is critical. Commonly used exploration strategies in reinforcement learning are largely undirected with sharp degradation as action dimensionality grows. Many existing methods resort to dimensionality reduction, which constrains policy expressiveness and forfeits system flexibility. We introduce Q-guided Flow Exploration (Qflex), a scalable reinforcement learning method that conducts exploration directly in the native high-dimensional action space. During training, Qflex traverses actions from a learnable source distribution along a probability flow induced by the learned value function, aligning exploration with task-relevant gradients rather than isotropic noise. Our proposed method substantially outperforms representative online reinforcement learning baselines across diverse high-dimensional continuous-control benchmarks. Qflex also successfully controls a full-body human musculoskeletal model to perform agile, complex movements, demonstrating superior scalability and sample efficiency in very high-dimensional settings. Our results indicate that value-guided flows offer a principled and practical route to exploration at scale.

💡 Research Summary

The paper tackles the long‑standing challenge of exploration in high‑dimensional continuous‑control problems, where the sheer size of the state‑action space renders traditional undirected exploration methods ineffective. The authors first formalize the “vanishing exploration” phenomenon: when isotropic Gaussian noise is added to each action dimension, the variance of the resulting end‑effector position scales as O(1/|A|), causing the exploratory reach to collapse as the number of degrees of freedom grows. This analysis, supported by a proof in the appendix, explains why standard Gaussian perturbations quickly become sample‑inefficient for systems with hundreds of actuators, such as full‑body musculoskeletal models.

To overcome this limitation, the authors introduce Q‑guided Flow Exploration (Q‑flex), a novel exploration mechanism that leverages the learned state‑action value function Q(s,a) to construct a directed probability flow in the native action space. The method builds on recent advances in flow‑matching generative modeling. A source distribution π₀θ(a|s), parameterized as a diagonal Gaussian, is transformed into a target distribution that concentrates probability mass on high‑value actions. The transformation is defined by an ordinary differential equation (ODE) da/dt = v(t,a), where the velocity field v(t,a) is set to M∇ₐQ(s,a). Here M is a positive‑definite preconditioner that can rescale and rotate the raw gradient for better conditioning. By integrating this ODE for N steps with step size η, each sampled action is progressively pushed toward regions of higher expected return.

The authors provide a theoretical guarantee: under mild smoothness assumptions on Q and boundedness of M, the expected value F(t;s)=E_{a∼π(t)}