Gaussian Processes for Sample Efficient Reinforcement Learning with RMAX-like Exploration

We present an implementation of model-based online reinforcement learning (RL) for continuous domains with deterministic transitions that is specifically designed to achieve low sample complexity. To achieve low sample complexity, since the environment is unknown, an agent must intelligently balance exploration and exploitation, and must be able to rapidly generalize from observations. While in the past a number of related sample efficient RL algorithms have been proposed, to allow theoretical analysis, mainly model-learners with weak generalization capabilities were considered. Here, we separate function approximation in the model learner (which does require samples) from the interpolation in the planner (which does not require samples). For model-learning we apply Gaussian processes regression (GP) which is able to automatically adjust itself to the complexity of the problem (via Bayesian hyperparameter selection) and, in practice, often able to learn a highly accurate model from very little data. In addition, a GP provides a natural way to determine the uncertainty of its predictions, which allows us to implement the “optimism in the face of uncertainty” principle used to efficiently control exploration. Our method is evaluated on four common benchmark domains.

💡 Research Summary

The paper introduces GP‑RMAX, a model‑based online reinforcement‑learning framework designed for continuous‑state domains with deterministic dynamics, aiming to achieve low sample complexity. The central contribution is the separation of function approximation in the model learner from the interpolation performed in the planner. For model learning, Gaussian‑process (GP) regression is employed. GPs are non‑parametric, automatically adapt their complexity via Bayesian hyper‑parameter optimization, and crucially provide a predictive variance that quantifies uncertainty. This uncertainty is leveraged to implement an “optimism in the face of uncertainty” exploration strategy, analogous to the RMAX algorithm, by treating poorly‑known state‑action pairs as maximally rewarding.

The algorithm consists of two interleaved modules. The model learner collects transition triples (state, action, next‑state) during interaction and, every K steps, updates a set of independent univariate GPs—one for each dimension of the state change Δx = x′‑x—using the accumulated dataset D. By modeling the change rather than the absolute next state, the input dimensionality remains constant and learning is more stable. The planner receives the current GP model ˆf and solves the Bellman equation on a uniform grid Γ_h over the state space using value iteration. Linear (or higher‑order B‑spline) interpolation is used to evaluate Q‑values at arbitrary points, and the Bellman operator is expressed compactly as T_Γh Q = R + γ·max_a′ W_a Q_a′, where W_a is a sparse interpolation matrix.

Exploration is driven by the GP’s predictive variance: regions with high variance are assigned the maximal possible reward (R_max) during planning, encouraging the agent to visit them. This yields a principled, sample‑efficient exploration policy without requiring explicit exploration bonuses.

Experiments were conducted on four standard continuous‑control benchmarks: Mountain Car, Pendulum, Acrobot, and a 2‑D navigation task. GP‑RMAX was compared against Fitted R‑MAX, LSPI, and model‑free deep methods such as DDPG. Results show that GP‑RMAX reaches the goal with far fewer environment interactions, especially during the early learning phase, and that its automatic hyper‑parameter tuning allows robust performance without domain‑specific tuning.

The authors acknowledge that the uniform‑grid approach suffers from the curse of dimensionality; thus the current method is practical for low‑dimensional (typically 2–3‑D) state spaces. Extending the approach to higher dimensions would require sparse or adaptive grids, or alternative function‑approximation schemes. Theoretical analysis of sample complexity is limited; the paper argues qualitatively that accurate transition models enable near‑optimal planning with minimal samples, but formal bounds are left for future work.

In summary, GP‑RMAX demonstrates that combining Gaussian‑process regression for model learning with RMAX‑style optimistic exploration yields a highly sample‑efficient reinforcement‑learning algorithm for deterministic continuous domains. The work opens avenues for further research on scaling to higher dimensions, handling stochastic dynamics, and deriving rigorous sample‑complexity guarantees.