Robustness to Model Approximation, Model Learning From Data, and Sample Complexity in Wasserstein Regular MDPs

The paper studies the robustness properties of discrete-time stochastic optimal control under Wasserstein model approximation for both discounted-cost and average-cost criteria. Specifically, we study the performance loss when applying an optimal policy designed for an approximate model to the true dynamics compared with the optimal cost for the true model under the sup-norm-induced metric, and relate it to the Wasserstein-1 distance between the approximate and true transition kernels. A primary motivation of this analysis is empirical model learning, as well as empirical noise distribution learning, where Wasserstein convergence holds under mild conditions but stronger convergence criteria, such as total variation, may not. We discuss applications of the results to the disturbance estimation problem, where sample complexity bounds are given, and also to a general empirical model learning approach, obtained under either Markov or i.i.d. learning settings.

💡 Research Summary

The paper investigates how approximation errors measured by the Wasserstein‑1 distance affect the performance of discrete‑time stochastic optimal control problems, under both discounted‑cost and average‑cost criteria. The authors first formalize the setting of a Markov decision process (MDP) with Polish state space, compact action space, bounded continuous cost, and weakly continuous transition kernel. They then derive Lipschitz continuity results for the optimal value functions with respect to perturbations in both the cost function and the transition kernel. Specifically, for the discounted case they prove an inequality of the form

‖Jβ*(c,T) – Jβ*(ĉ,Ť)‖∞ ≤ Lβ (‖c – ĉ‖∞ + K·W1(T,Ť)),

where Lβ depends on the discount factor β, K is the Lipschitz constant of the kernel in the Wasserstein‑1 metric, and W1 denotes the Wasserstein‑1 distance between the true and approximate kernels. For the average‑cost case they provide two parallel approaches: one based on a minorization condition that guarantees a regeneration structure, and another that takes the limit of the discounted result as β → 1 (the vanishing‑discount method). Both yield analogous Lipschitz bounds, albeit with different additional regularity assumptions.

Using these continuity results, the paper defines a “robustness error” – the performance loss incurred when the optimal policy of an approximate model (ĉ,Ť) is applied to the true model (c,T). The error is bounded above by a constant times the same combination of cost and kernel discrepancies, giving a clear quantitative relationship between model accuracy and control performance. This relationship enables practitioners to set explicit accuracy requirements on learned models before deploying the resulting policies.

The second major contribution is a statistical analysis that translates the deterministic robustness bounds into sample‑complexity guarantees for model learning from data. Two data‑generation scenarios are considered: (a) data collected along a controlled trajectory (i.e., a single Markov chain generated by a possibly sub‑optimal policy) and (b) i.i.d. samples obtained from a simulator of the transition kernel. For each case the authors construct empirical estimates of the transition kernel (ŤN) and cost function (ĉN) and apply the robustness bounds to the expected robustness error. Under standard mixing or geometric ergodicity assumptions for the controlled trajectory, they show that the Wasserstein‑1 distance between ŤN and T converges at a rate O(N^{-1/2}) with high probability, leading to an O(N^{-1/2}) decay of the robustness error. For the i.i.d. setting the same rate follows from classical concentration results for empirical measures in Wasserstein distance.

A further refinement is obtained when the state space is discretized via a fixed quantization grid of mesh size h. Assuming the true transition kernel is Lipschitz in the Wasserstein sense, the quantization error contributes an additional term proportional to h. By balancing h against the sample size N, the authors derive overall error rates of order O(N^{-1/(d+2)}) for a d‑dimensional state space, which matches known non‑parametric rates for static discretization of MDPs.

The final part of the paper treats disturbance‑distribution learning. For systems of the form X_{t+1}=f(X_t,U_t,W_t) with i.i.d. disturbance W_t ∼ μ, the authors consider replacing μ by an empirical distribution ν̂_N. Using the same Wasserstein‑based robustness analysis, they obtain error bounds that decay as O(N^{-1/2}) under mild regularity of μ, and improve to parametric O(N^{-1}) rates when μ satisfies additional uniform regularity conditions. This section shows that disturbance‑estimation can be viewed as a special case of model approximation, and the earlier theorems apply directly.

Overall, the paper unifies and extends prior work on robustness to model misspecification by focusing on Wasserstein‑1 convergence, which is weaker than total‑variation but strong enough to guarantee meaningful performance guarantees. It provides a comprehensive theoretical framework that links model approximation quality, policy robustness, and statistical sample complexity for both discounted and average‑cost MDPs, and it offers concrete guidance for practitioners designing data‑driven control policies.