Intrinsic Reward Policy Optimization for Sparse-Reward Environments

Exploration is essential in reinforcement learning as an agent relies on trial and error to learn an optimal policy. However, when rewards are sparse, naive exploration strategies, like noise injection, are often insufficient. Intrinsic rewards can also provide principled guidance for exploration by, for example, combining them with extrinsic rewards to optimize a policy or using them to train subpolicies for hierarchical learning. However, the former approach suffers from unstable credit assignment, while the latter exhibits sample inefficiency and sub-optimality. We propose a policy optimization framework that leverages multiple intrinsic rewards to directly optimize a policy for an extrinsic reward without pretraining subpolicies. Our algorithm – intrinsic reward policy optimization (IRPO) – achieves this by using a surrogate policy gradient that provides a more informative learning signal than the true gradient in sparse-reward environments. We demonstrate that IRPO improves performance and sample efficiency relative to baselines in discrete and continuous environments, and formally analyze the optimization problem solved by IRPO. Our code is available at https://github.com/Mgineer117/IRPO.

💡 Research Summary

The paper tackles the long‑standing challenge of learning in sparse‑reward reinforcement learning (RL) environments, where conventional exploration methods such as action‑space noise or parameter‑space perturbations often fail to generate informative gradients. Existing intrinsic‑reward approaches either blend intrinsic and extrinsic signals—leading to unstable credit assignment when the intrinsic term dominates—or rely on hierarchical RL with pretrained sub‑policies, which introduces sample inefficiency and limits fine‑grained decision making.

To overcome these limitations, the authors introduce Intrinsic Reward Policy Optimization (IRPO), a novel framework that leverages multiple intrinsic reward functions to drive exploration, then directly uses the extrinsic‑reward gradients of the resulting exploratory policies to update a single base policy. The key steps are:

1. Multiple Intrinsic Rewards – Define K intrinsic reward functions (\tilde R_k). For each, create an exploratory policy (\tilde\pi_k) initialized as a copy of the current base policy (\pi_\theta).

2. Exploratory Updates – Each exploratory policy is trained for N gradient steps using only its associated intrinsic reward. Two critics are maintained per policy: an intrinsic critic ( \tilde V_\phi^k) for estimating the intrinsic value, and an extrinsic critic ( V_\phi^k) for estimating the extrinsic value of the same policy. The intrinsic critic guides the N updates, while the extrinsic critic records how much true task reward the policy would obtain after those updates.

3. Gradient Back‑Propagation – After the N intrinsic updates, the algorithm computes the extrinsic‑reward policy gradient (\nabla_{\tilde\theta_k} J_R(\tilde\theta_k)) for each exploratory policy using the extrinsic critic. To propagate this information back to the base parameters (\theta), the Jacobians of each intermediate update (\partial \tilde\theta^{(j)}k / \partial \tilde\theta^{(j+1)}k) are stored. The chain rule yields (\nabla\theta J_R(\tilde\theta_k) = \big(\prod{j=1}^N \partial \tilde\theta^{(j)}_k / \partial \tilde\theta^{(j+1)}k\big)^\top \nabla{\tilde\theta_k} J_R(\tilde\theta_k)).

4. Weighted Aggregation – The contributions of the K exploratory policies are combined with soft‑max weights (\omega_k = \exp(J_R(\tilde\theta_k)/\tau) / \sum_{k’} \exp(J_R(\tilde\theta_{k’})/\tau)). The temperature (\tau) is annealed toward zero, eventually focusing the update on the exploratory policy that achieved the highest extrinsic return.

5. Base Policy Update – The aggregated “IRPO gradient” (\nabla J_{\text{IRPO}}(\theta) = \sum_{k=1}^K \omega_k \nabla_\theta J_R(\tilde\theta_k)) is used in a trust‑region (TRPO‑style) optimization step to ensure stable policy changes.

The authors provide a formal analysis showing that in sparse‑reward settings the true policy gradient (|\nabla_\theta J(\theta)|2) vanishes as reward sparsity increases (Corollary 3.1). This justifies the need for a surrogate gradient. They also define the reachable set of exploratory policies (\tilde\Gamma_N) and prove that, with (\tau \to 0), IRPO solves the optimization problem (\max{\theta} \max_{k} J_R(\tilde\theta_k^{(N+1)}(\theta))), effectively selecting the base policy whose N‑step intrinsic updates lead to the best extrinsic performance.

Empirical evaluation spans both discrete (MiniGrid) and continuous (MuJoCo) benchmarks. Baselines include: (i) action‑space noise, (ii) parameter‑space noise, (iii) uncertainty‑based intrinsic rewards (state‑visit counts, entropy bonuses, predictive error), and (iv) hierarchical RL with pretrained sub‑policies. Across all tasks, IRPO achieves higher average returns with fewer environment interactions. Notably, in the early training phase—where extrinsic signals are extremely rare—IRPO’s exploratory policies discover rewarding states far more efficiently than the baselines.

Ablation studies examine the impact of the number of intrinsic rewards K, the number of intrinsic updates N, the temperature schedule, and the approximation of Jacobians. Results indicate that performance is robust to these hyper‑parameters and that alternative intrinsic rewards (e.g., information‑gain, random‑network‑distillation) can replace the diffusion‑maximizing reward used in the main experiments without degrading the core benefit.

In summary, IRPO introduces a principled way to harness intrinsic rewards for exploration while preserving a clean, direct optimization of the extrinsic objective. It eliminates the credit‑assignment instability of reward‑mixing methods, avoids the sample‑heavy pretraining of hierarchical approaches, and delivers superior sample efficiency and final performance in sparse‑reward domains. Future directions suggested include automated discovery of intrinsic reward functions, multi‑step chaining of exploratory policies, and scaling the framework to multi‑agent or large‑scale real‑world tasks.