A Theoretical Analysis of State Similarity Between Markov Decision Processes

The bisimulation metric (BSM) is a powerful tool for analyzing state similarities within a Markov decision process (MDP), revealing that states closer in BSM have more similar optimal value functions. While BSM has been successfully utilized in reinforcement learning (RL) for tasks like state representation learning and policy exploration, its application to state similarity between multiple MDPs remains challenging. Prior work has attempted to extend BSM to pairs of MDPs, but a lack of well-established mathematical properties has limited further theoretical analysis between MDPs. In this work, we formally establish a generalized bisimulation metric (GBSM) for measuring state similarity between arbitrary pairs of MDPs, which is rigorously proven with three fundamental metric properties, i.e., GBSM symmetry, inter-MDP triangle inequality, and a distance bound on identical spaces. Leveraging these properties, we theoretically analyze policy transfer, state aggregation, and sampling-based estimation across MDPs, obtaining explicit bounds that are strictly tighter than existing ones derived from the standard BSM. Additionally, GBSM provides a closed-form sample complexity for estimation, improving upon existing asymptotic results based on BSM. Numerical results validate our theoretical findings and demonstrate the effectiveness of GBSM in multi-MDP scenarios.

💡 Research Summary

The paper introduces a Generalized Bisimulation Metric (GBSM) that quantifies state similarity across arbitrary pairs of Markov Decision Processes (MDPs). While the classic bisimulation metric (BSM) is defined only within a single MDP and enjoys pseudometric properties (symmetry, triangle inequality, indiscernibility), extending it to inter‑MDP comparisons has been problematic. The authors define GBSM by first constructing a state‑action discrepancy δ(d) = |R₁(s,a) – R₂(s′,a′)| + γ·W₁(P₁(·|s,a), P₂(·|s′,a′); d) and then applying the Hausdorff distance between the action sets Xₛ and Xₛ′ of the two states. They prove existence of a unique fixed‑point distance via a contraction argument (Theorem 1) and show that this distance upper‑bounds the difference between the optimal value functions of the two MDPs (Theorem 2).

Three fundamental metric properties are rigorously established: (i) symmetry (Theorem 3) – the distance is unchanged when the order of the MDPs is swapped; (ii) inter‑MDP triangle inequality (Theorem 4) – for any three MDPs, d₁₋₂ ≤ d₁₋₃ + d₃₋₂, proved using the Gluing Lemma for Wasserstein distances; (iii) a bound on identical spaces (Theorem 5) – when the state and action spaces coincide, the maximal self‑distance is bounded by (1/(1‑γ))·maxₛ H(Xₛ,Xₛ;δ_TV), which becomes zero if the two MDPs are identical, thus recovering indiscernibility.

Leveraging these properties, the paper derives tighter theoretical results for three key RL scenarios. For policy transfer, Theorem 6 gives a regret bound that combines the GBSM between source and target MDPs with the source‑policy regret, improving upon prior BSM‑based bounds that only considered one‑step effects. For state aggregation, Theorem 7 provides a value‑function approximation error directly proportional to the GBSM between aggregated states, yielding strictly smaller errors than existing BSM‑based analyses. For sampling‑based estimation, Theorems 8 and 9 present an explicit, closed‑form sample‑complexity expression O((1‑γ)⁻²·ε⁻²·log(1/δ)), surpassing earlier asymptotic results that lacked concrete constants.

Empirical validation is performed on random Garnet MDPs and a simulation‑to‑real wireless‑network RL task. Results confirm that GBSM yields more accurate state similarity estimates, tighter performance guarantees, and requires fewer samples than standard BSM. The authors also demonstrate that GBSM naturally extends to variants such as lax BSM and on‑policy BSM, and they propose an efficient algorithm that combines aggregation and estimation for large‑scale problems.

Overall, the work fills a critical gap in multi‑MDP theory by providing a rigorously defined, metrically sound similarity measure, and by showing how this measure can be exploited to obtain stronger guarantees in policy transfer, state abstraction, and sample‑efficient learning.