Analysis of Control Bellman Residual Minimization for Markov Decision Problem

Markov decision problems are most commonly solved via dynamic programming. Another approach is Bellman residual minimization, which directly minimizes the squared Bellman residual objective function. However, compared to dynamic programming, this approach has received relatively less attention, mainly because it is often less efficient in practice and can be more difficult to extend to model-free settings such as reinforcement learning. Nonetheless, Bellman residual minimization has several advantages that make it worth investigating, such as more stable convergence with function approximation for value functions. While Bellman residual methods for policy evaluation have been widely studied, methods for policy optimization (control tasks) have been scarcely explored. In this paper, we establish foundational results for the control Bellman residual minimization for policy optimization.

💡 Research Summary

The paper investigates the theoretical foundations of Control Bellman Residual (CBR) minimization for policy optimization in Markov Decision Processes (MDPs). While Bellman residual methods have been extensively studied for policy evaluation, their use in control (i.e., finding an optimal policy) has received little attention. The authors fill this gap by defining a squared‑error objective f(θ)=½‖T(Φθ)−Φθ‖², where T is the standard control Bellman operator (including a max over actions), Φθ is a linear function approximator of the Q‑function, and θ are the parameters.

Key structural results:

1. Piecewise‑quadratic nature – Because of the max operator, the parameter space decomposes into regions Sπ associated with each deterministic greedy policy π. Within each region the objective reduces to a single quadratic function, making the problem locally convex and smooth. The regions are shown to be homogeneous half‑spaces and convex cones, providing a clear geometric picture of where policy switches occur.
2. Quadratic bounds – The authors derive tight lower and upper quadratic bounds q₁(θ) ≤ f(θ) ≤ q₂(θ). Both bounds share the same minimizer, namely the parameter vector that minimizes the Euclidean distance to the true optimal Q‑function Q*. This links CBR minimization directly to the quality of the underlying Q‑approximation.

Non‑smooth analysis:
To handle non‑differentiability, the paper computes the Clarke sub‑differential of f:

∂f(θ)= { Φᵀ(γ P Π_β − I)ᵀ (TQθ − Qθ) | β ∈ conv(Λ(Qθ)) }

where Λ(Q) is the set of greedy policies for a given Q‑function and conv denotes the convex hull. This expression reveals that any stationary point satisfies an oblique projection condition: the Bellman error lies in the range of (γ P Π_β − I)Φ and is orthogonal to its complement. The authors formalize this as the Oblique‑Projected Control Bellman Equation (OP‑CBE):

Qθ = Γ_{Φ|Ψ_β} TQθ, Ψ_β = (γ P Π_β − I)Φ

where Γ is the oblique projector onto the column space of Φ along the column space of Ψ_β. This generalizes the orthogonal projection used in policy evaluation and shows that a stationary solution always exists, even when the associated operator is not a contraction.

Algorithmic contribution:
Because ∂f(θ) is set‑valued, the authors adopt a generalized gradient descent scheme that selects the minimum‑norm subgradient at each iteration:

θ_{k+1}=θ_k − α_k g_k, g_k ∈ arg min_{g∈∂f(θ_k)}‖g‖₂

Step sizes α_k are chosen by a back‑tracking line search satisfying the Armijo condition. Leveraging results from Burke et al. (2005), they prove that any limit point of this process is a stationary point (0 ∈ ∂f).

Soft Bellman residual:
To obtain a smooth objective, the paper introduces the Soft Control Bellman Residual (SCBR) using a soft‑max operator F_λ with temperature λ>0:

f_s(θ)=½‖F_λ(Φθ)−Φθ‖²

Since F_λ is differentiable, standard gradient descent can be applied. The authors show that SCBR retains the piecewise‑quadratic structure and Lipschitz continuity, and they prove convergence of vanilla gradient descent under standard step‑size rules.

Empirical evaluation:
Experiments on benchmark MDPs with linear function approximation compare CBR/SCBR‑based optimization against Projected Value Iteration (PVI). Results indicate that the residual‑based methods converge faster to a near‑optimal policy and are more robust to function‑approximation errors, especially when the discount factor γ is close to 1. SCBR, in particular, exhibits smoother learning curves due to the differentiable soft‑max.

Implications and future work:
The study establishes that non‑convex, non‑smooth Bellman residual minimization can be rigorously analyzed and efficiently optimized using Clarke sub‑differentials and oblique projection theory. This provides a new avenue for stable policy optimization in settings where traditional dynamic programming loses its contraction property (e.g., with function approximation or off‑policy data). Future directions include extending the framework to deep neural network approximators, multi‑agent settings, and developing sample‑efficient double‑sampling or off‑policy correction techniques.

Overall, the paper delivers a solid theoretical foundation for control‑oriented Bellman residual minimization and demonstrates its practical viability as an alternative to classic DP‑based control algorithms.