Approximating the Uniform Value in Hidden Stochastic Games with Doeblin Conditions

In \emph{zero-sum two-player hidden stochastic games}, players observe partial information about the state. We address: $(i)$ the existence of the \emph{uniform value}, i.e., a limiting average payoff that both players can guarantee for sufficiently long durations, and $(ii)$ the existence of an algorithm to approximate it. Previous work shows that, in the general case, the uniform value may fail to exist, and, even when it does, there need not exist an algorithm to compute or approximate it. Therefore, we consider the \emph{Doeblin condition} in hidden stochastic games, requiring that, after a sufficiently long time, the posterior beliefs have a uniformly positive probability of resetting to one of finitely many neighborhoods in the belief space. We prove the existence of the uniform value and provide an algorithm to approximate it. We identify sufficient conditions, namely \emph{ergodicity} in the blind setting (when the signal is uninformative) and \emph{primitivity} in the hidden setting (when there are multiple signals). Moreover, we show that, in the hidden setting, ergodicity does not guarantee the Doeblin condition. Our results are new even for the one-player setting, i.e., partially observable Markov decision processes.

💡 Research Summary

The paper tackles two fundamental questions for zero‑sum two‑player hidden stochastic games (HSGs): does a uniform value exist, and can it be approximated algorithmically? In fully observable stochastic games, Mertens and Neyman proved the existence of a uniform value and subsequent works provided practical algorithms. However, when players receive only partial information through signals, the situation changes dramatically. Ziliotto constructed HSGs where the uniform value fails to exist even with uninformative signals, and Madani et al. showed that even when it exists, computing or approximating it is undecidable. Consequently, a subclass of HSGs with both existence and algorithmic approximability is highly desirable.

The authors introduce a Doeblin condition for HSGs, inspired by the classical Doeblin condition for Markov chains. Roughly, after a sufficiently long number of stages, the posterior belief (the common probability distribution over states) has a uniformly positive probability ε of falling into one of finitely many pre‑specified neighborhoods in the belief simplex. This “reset” property guarantees that the belief process cannot drift arbitrarily far without repeatedly being pulled back into a finite set of regions.

Using this condition, the paper establishes two main results. First, Theorem 3.1 proves that any Doeblin HSG possesses a uniform value. The proof proceeds by discretizing the belief space into a finite partition, thereby defining an abstract stochastic game with a finite state space. The original (infinite‑belief) game and its abstract counterpart are coupled block‑wise: each block is split into a “reset” sub‑block where the Doeblin condition forces the beliefs of the two processes to become close with high probability, followed by a “steady” sub‑block where the same action pair is played in both games. By carefully choosing the lengths of these sub‑blocks, the loss incurred during the reset phase becomes negligible, and the average payoffs of the two games stay arbitrarily close for any horizon n. Since finite‑state stochastic games always have a uniform value (Mertens & Neyman, 1981), the limit of the abstract game’s uniform values coincides with the uniform value of the original HSG.

Second, Theorem 3.2 provides an explicit algorithm to approximate the uniform value of a Doeblin HSG. The algorithm enumerates a sufficiently fine discretization of the belief simplex, constructs the corresponding abstract game, and computes its uniform value using known polynomial‑time methods (e.g., the algorithm of Oliu‑Barton et al., 2021). The approximation error can be bounded as a function of the discretization granularity and the Doeblin constant ε, yielding a controllable trade‑off between computational effort and precision.

Beyond the abstract condition, the authors identify concrete structural sufficient conditions that guarantee the Doeblin property. In the blind setting (a single, uninformative signal), ergodicity—the ability of any two initial beliefs to become ε‑close after a sufficiently long sequence of action pairs—implies the Doeblin condition (Theorem 3.5). In the hidden setting with multiple signals, primitivity of the set of transition‑signal matrices (i.e., there exists a power of the combined matrix with all entries strictly positive) ensures the Doeblin reset (Theorem 3.7). These results connect the new condition to well‑studied concepts in Markov chain theory and provide practical criteria for modelers.

Importantly, the paper also demonstrates the limits of these criteria. Extending ergodicity naïvely to the hidden case does not guarantee the Doeblin property; a constructed three‑state, three‑signal HSG satisfies an ergodic mixing condition yet fails to possess a uniform value (Theorem 6.3). This negative example underscores that belief dynamics in partially observable environments can retain memory in ways that ergodicity alone cannot eliminate.

The contributions are threefold. First, it is the first work to employ a Doeblin‑type reset condition to prove both existence and approximability of the uniform value in hidden stochastic games. Second, it delineates a clear computational boundary: while exact computation of the uniform value is decidable for fully observable stochastic games, and undecidable in general HSGs, the uniform value remains undecidable even within the Doeblin subclass, highlighting that Doeblin HSGs are strictly more complex than their fully observable counterparts. Third, because many ω‑regular objectives (e.g., reachability, safety) can be expressed via the uniform value, the results immediately extend to a broad class of verification and synthesis problems under partial observation.

Overall, the paper establishes a robust theoretical framework that bridges stochastic game theory, Markov chain ergodicity, and algorithmic game theory, opening new avenues for the analysis and computation of long‑run equilibria in environments where information is inherently hidden.