Proximal Policy Optimization in Path Space: A Schrödinger Bridge Perspective

On-policy reinforcement learning with generative policies is promising but remains underexplored. A central challenge is that proximal policy optimization (PPO) is traditionally formulated in terms of action-space probability ratios, whereas diffusion- and flow-based policies are more naturally represented as trajectory-level generative processes. In this work, we propose GSB-PPO, a path-space formulation of generative PPO inspired by the Generalized Schrödinger Bridge (GSB). Our framework lifts PPO-style proximal updates from terminal actions to full generation trajectories, yielding a unified view of on-policy optimization for generative policies. Within this framework, we develop two concrete objectives: a clipping-based objective, GSB-PPO-Clip, and a penalty-based objective, GSB-PPO-Penalty. Experimental results show that while both objectives are compatible with on-policy training, the penalty formulation consistently delivers better stability and performance than the clipping counterpart. Overall, our results highlight path-space proximal regularization as an effective principle for training generative policies with PPO.

💡 Research Summary

This paper addresses a key gap in on‑policy reinforcement learning (RL) when using expressive generative policies such as diffusion models or flow‑based generators. Traditional Proximal Policy Optimization (PPO) operates on action‑space probability ratios, which works well for simple Gaussian actors but becomes ill‑defined for policies that generate actions through multi‑step stochastic processes. The authors propose a path‑space formulation of PPO, called GSB‑PPO, inspired by the Generalized Schrödinger Bridge (GSB) framework, which treats stochastic generation as an optimization over entire trajectory distributions rather than merely the terminal marginal.

The core idea is to view a generative policy as a stochastic differential equation (SDE) or its discrete reverse‑diffusion chain, yielding a trajectory distribution (P_\theta(\mathbf{a}_{0:N}\mid s)). The executed action is the final denoised sample (a = \mathbf{a}_0). Because the RL objective depends only on this terminal action, any advantage term can be expressed equivalently under the full path distribution. This observation enables the lifting of the standard PPO surrogate \