Governing Strategic Dynamics: Equilibrium Stabilization via Divergence-Driven Control

Black-box coevolution in mixed-motive games is often undermined by opponent-drift non-stationarity and noisy rollouts, which distort progress signals and can induce cycling, Red-Queen dynamics, and detachment. We propose the \emph{Marker Gene Method} (MGM), a curriculum-inspired governance mechanism that stabilizes selection by anchoring evaluation to cross-generational marker individuals, together with DWAM and conservative marker-update rules to reduce spurious updates. We also introduce NGD-Div, which adapts the key update threshold using a divergence proxy and natural-gradient optimization. We provide theoretical analysis in strictly competitive settings and evaluate MGM integrated with evolution strategies (MGM-E-NES) on coordination games and a resource-depletion Markov game. MGM-E-NES reliably recovers target coordination in Stag Hunt and Battle of the Sexes, achieving final cooperation probabilities close to $(1,1)$ (e.g., $0.991\pm0.01/1.00\pm0.00$ and $0.97\pm0.00/0.97\pm0.00$ for the two players). In the Markov resource game, it maintains high and stable state-conditioned cooperation across 30 seeds, with final cooperation of $\approx 0.954/0.980/0.916$ in \textsc{Rich}/\textsc{Poor}/\textsc{Collapsed} (both players; small standard deviations), indicating welfare-aligned and state-dependent behavior. Overall, MGM-E-NES transfers across tasks with minimal hyperparameter changes and yields consistently stable training dynamics, showing that top-level governance can substantially improve the robustness of black-box coevolution in dynamic environments.

💡 Research Summary

The paper tackles a fundamental instability in black‑box co‑evolutionary learning for mixed‑motive games: the non‑stationarity of fitness evaluation caused by opponent drift, noisy rollouts, and the resulting Red‑Queen, intransitivity, and detachment pathologies. To address this, the authors introduce the Marker Gene Method (MGM), a curriculum‑inspired governance layer that anchors evaluation to a cross‑generational “marker” individual. Each generation, candidate policies are evaluated against the current marker rather than only against the evolving opponent population, providing a stable reference point even as the opponent drifts. The marker itself is only updated when a dynamic‑weighting criterion (DWAM) signals that the marker is sufficiently outperformed by the population. This “specialize → generalize → update” schedule prevents premature or overly frequent marker changes, thereby reducing spurious fitness signals.

A key hyper‑parameter of MGM is the DWAM threshold λ, which determines when to replace the marker. Since λ depends on game scale and equilibrium structure, the authors propose NGD‑Div, a lightweight natural‑gradient controller that adapts λ online using a divergence proxy (typically the KL divergence between the current policy and the marker). NGD‑Div increases λ when the divergence is below a target and decreases it when the divergence exceeds the target, effectively keeping the marker update regime in a stable operating region without manual tuning.

Theoretical analysis is performed in symmetric strictly competitive games (SCGs), where the payoff matrix can be transformed into an antisymmetric form. Under bounded payoffs, finite‑sample evaluation, and weak dependence assumptions, the authors prove a separation‑of‑time‑scales result: entering a Nash‑equilibrium neighbourhood occurs on a fast time‑scale, while escaping it under stochastic perturbations occurs on a much slower time‑scale. Specifically, the expected escape time grows exponentially with the inverse of the perturbation magnitude, implying that once the system reaches a stable equilibrium it will remain there for an exponentially long period.

Empirically, the authors instantiate MGM with Natural Evolution Strategies (NES), forming MGM‑E‑NES, and evaluate it on several benchmarks: (1) coordination games (Stag Hunt and Battle of the Sexes), where MGM‑E‑NES achieves near‑perfect cooperation probabilities (≈0.99–1.00 for both players); (2) high‑dimensional Rock‑Paper‑Scissors (3‑D, 100‑D, 1000‑D) competitive games, where final KL divergence is essentially zero and convergence is rapid; (3) a resource‑depletion Markov game with three environmental states (Rich, Poor, Collapsed). In the Markov game, MGM‑E‑NES maintains high, state‑conditioned cooperation across 30 random seeds (≈0.954, 0.980, 0.916 respectively) with very low variance, demonstrating welfare‑aligned, state‑dependent behavior.

A comprehensive comparison against baselines—Optimistic Gradient Descent Ascent (OGDA), Policy‑Space Response Oracles (PSRO), Pure NES, LOLA, Fixed‑Point (FP), and a simple DPG variant—shows that MGM‑E‑NES consistently outperforms in both final solution quality and training stability. OGDA converges quickly but often to sub‑optimal mixed strategies; PSRO can achieve good performance but incurs large expansion costs; Pure NES and LOLA suffer from high variance and occasional collapse.

The paper’s contributions are threefold: (1) the design of a marker‑gene based governance mechanism with bounded rollback that stabilizes evaluation in black‑box co‑evolution; (2) a theoretical separation‑of‑time‑scales analysis proving exponential escape times from equilibria under modest noise; (3) the NGD‑Div controller that automatically tunes the DWAM threshold, making the approach largely hyper‑parameter free. By decoupling the governance layer from the underlying actuator, the framework is compatible with any black‑box optimizer, not just NES.

In summary, “governance”—the introduction of an external, slowly‑evolving reference point—proves to be a powerful tool for mitigating non‑stationarity in co‑evolutionary learning. The combination of MGM and NGD‑Div yields a robust, transferable method that delivers stable training dynamics and high‑quality equilibria across a diverse set of games, with minimal manual tuning. This work opens a promising direction for future research on hierarchical control and curriculum‑style stabilization in multi‑agent reinforcement learning and evolutionary computation.