Provably Optimal Reinforcement Learning under Safety Filtering

Recent advances in reinforcement learning (RL) enable its use on increasingly complex tasks, but the lack of formal safety guarantees still limits its application in safety-critical settings. A common practical approach is to augment the RL policy with a safety filter that overrides unsafe actions to prevent failures during both training and deployment. However, safety filtering is often perceived as sacrificing performance and hindering the learning process. We show that this perceived safety-performance tradeoff is not inherent and prove, for the first time, that enforcing safety with a sufficiently permissive safety filter does not degrade asymptotic performance. We formalize RL safety with a safety-critical Markov decision process (SC-MDP), which requires categorical, rather than high-probability, avoidance of catastrophic failure states. Additionally, we define an associated filtered MDP in which all actions result in safe effects, thanks to a safety filter that is considered to be a part of the environment. Our main theorem establishes that (i) learning in the filtered MDP is safe categorically, (ii) standard RL convergence carries over to the filtered MDP, and (iii) any policy that is optimal in the filtered MDP-when executed through the same filter-achieves the same asymptotic return as the best safe policy in the SC-MDP, yielding a complete separation between safety enforcement and performance optimization. We validate the theory on Safety Gymnasium with representative tasks and constraints, observing zero violations during training and final performance matching or exceeding unfiltered baselines. Together, these results shed light on a long-standing question in safety-filtered learning and provide a simple, principled recipe for safe RL: train and deploy RL policies with the most permissive safety filter that is available.

💡 Research Summary

The paper tackles a fundamental question in safe reinforcement learning (RL): does the use of a safety filter inevitably compromise the asymptotic performance of the learned policy? By introducing two complementary formalizations—a safety‑critical Markov decision process (SC‑MDP) that enforces categorical avoidance of failure states, and a filtered MDP (M_φ) that embeds a perfect, least‑restrictive safety filter into the environment—the authors prove that the answer is “no.”

In the SC‑MDP, the failure set F must never be entered; safety is guaranteed by restricting the policy to the maximal controlled‑invariant safe set Ω*. The admissible policy set Π_safe consists of all stochastic kernels whose support lies within the safe action set A_safe(s) for every s∈Ω*. The filtered MDP, on the other hand, treats the safety filter φ as part of the dynamics: any proposed action a is replaced by φ(s,a), which is guaranteed to be safe, and the transition and reward functions are defined with respect to this corrected action. Crucially, φ is defined to be “perfect”: it intervenes only when the nominal action would immediately leave Ω*, and otherwise leaves the action untouched.

The main theorem has three parts. First, standard RL convergence results (e.g., for Q‑learning, policy gradient, actor‑critic methods) carry over unchanged to M_φ because the filter does not break the Markov property, boundedness, or discounting assumptions. Second, any policy learned in M_φ, when executed with the same filter at deployment, satisfies the categorical safety constraint of the SC‑MDP; formally, π∘φ∈Π_safe. Third, the optimal value function of M_φ coincides with that of the SC‑MDP, implying that a policy π* that is optimal in the filtered environment also attains the maximal possible expected return among all safe policies in the original SC‑MDP. The proof hinges on the maximality of Ω*: because φ only modifies actions that would leave Ω*, the distribution over safe trajectories under π∘φ is identical to the distribution under π in M_φ, establishing value equivalence.

These results overturn the widely held belief that safety filtering necessarily introduces a performance penalty. The trade‑off is shown to be an artifact of overly conservative filters; a minimally restrictive, “bubble‑wrapped” filter enables the agent to focus solely on reward maximization while safety is guaranteed automatically.

Empirically, the authors validate the theory on Safety Gymnasium, a modern benchmark suite extending OpenAI’s Safety Gym. They train state‑of‑the‑art RL algorithms (PPO, SAC) inside the filtered environment across multiple tasks (goal reaching, obstacle avoidance) and constraints (speed limits, collision avoidance). Across all experiments, the training process incurs zero safety violations, and the final average returns match or exceed those of unfiltered baselines. Moreover, the proportion of steps where the filter intervenes is low, indicating that the filter is indeed permissive and does not hinder exploration.

In summary, the paper makes three key contributions: (1) a rigorous formal link between safety‑critical MDPs and filtered MDPs, (2) a proof that a perfect, least‑restrictive safety filter preserves both safety and asymptotic optimality, and (3) empirical evidence that the theoretical guarantees hold in realistic, high‑dimensional environments. The work provides a clear, principled recipe for safe RL: train any standard RL algorithm in a “bubble‑wrapped” environment using the most permissive safety filter available, then deploy the same policy with the same filter. This decouples safety enforcement from performance optimization, opening the door for safe RL in safety‑critical domains such as autonomous driving, robotics, and healthcare.