Nested Extremum Seeking Converges to Stackelberg Equilibrium

The nested Extremum Seeking (nES) algorithm is a model-free optimization method that has been shown to converge to a neighborhood of a Nash equilibrium. In this work, we demonstrate that the same nES dynamics can instead be made to converge to a neighborhood of a Stackelberg (leader–follower) equilibrium by imposing a different scaling law on the algorithm’s design parameters. For the two–level nested case, using Lie–bracket averaging and singular perturbation arguments, we provide a rigorous stability proof showing semi-global practical asymptotic convergence to a Stackelberg equilibrium under appropriate time-scale separation. The results reveal that equilibrium selection, Nash versus Stackelberg, depends not on modifying the closed-loop dynamics, but on the hierarchical scaling of design parameters and the induced time-scale structure. We demonstrate this effect using a simple quadratic example and the canonical Fish War game. The Stackelberg variant of nES provides a model-free framework for hierarchical optimization in multi-time-scale systems, with potential applications in power grids, networked dynamical systems, and tuning of particle accelerators.

💡 Research Summary

The paper investigates how the nested Extremum Seeking (nES) algorithm, originally known for model‑free convergence to a Nash equilibrium (NE), can be re‑engineered to converge to a Stackelberg equilibrium (SE) by merely altering the scaling of its design parameters. The authors focus on the two‑level (n = 2) nesting, which consists of a “leader” ES loop (slow) and a “follower” ES loop (fast). By choosing the follower’s dither frequency ω₂ and gain product α₂k₂ to be much larger than the leader’s ω₁ and α₁k₁, a clear separation of time scales is created.

The analysis proceeds in three stages. First, Lie‑bracket averaging is applied to the fast loop, eliminating the high‑frequency dither and yielding an averaged dynamics in which the follower follows a gradient descent on its own cost J_F with respect to x₂ while the leader’s state x₁ appears as a slowly varying parameter. Second, the authors introduce ε = 1/(α₂k₂) as a small singular‑perturbation parameter and rewrite the averaged system in standard singular‑perturbation form. The fast subsystem (boundary‑layer model) is shown to be exponentially stable under a strong convexity assumption on J_F (∂²J_F/∂x₂² ≥ m₂ > 0). This guarantees that the follower state rapidly converges to the unique solution h(x₁) of ∂J_F/∂x₂ = 0, i.e., the best‑response map. Third, with the follower effectively on its quasi‑steady‑state manifold x₂ ≈ h(x₁), the slow leader dynamics reduce to a single‑variable extremum‑seeking system whose cost is the reduced function (\tilde J_L(x₁)=J_L(x₁,h(x₁))). A second Lie‑bracket averaging step yields the averaged slow dynamics (\dot{\bar x}1 = -\alpha_1 k_1^2 \partial{x_1}\tilde J_L(\bar x_1)). Under a strong convexity assumption on (\tilde J_L) (∂²\tilde J_L/∂x₁² ≥ m₁ > 0), Lemma 1 guarantees global exponential stability of the equilibrium x₁* that minimizes (\tilde J_L).

The convergence proof hinges on two established results: (i) Theorem 1 (Lie‑bracket averaging) ensures the original nES trajectories stay arbitrarily close to the partially‑averaged system for sufficiently large ω₂; (ii) Theorem 2 (CTP + GUAS ⇒ SPUAS) states that if the averaged reduced system is globally uniformly asymptotically stable (GUAS) and the original and averaged systems satisfy the Converging Trajectories Property (CTP), then the original system is semi‑global practical uniformly asymptotically stable (SPUAS). Consequently, with ω₂ large enough and ε small enough (i.e., α₂k₂ large), the full nES dynamics converge to a neighborhood of the Stackelberg equilibrium ((x_1^,x_2^=h(x_1^*))).

Two illustrative examples validate the theory. A simple quadratic game with costs (J_L=(x_1-1)^2+(x_2-2)^2) and (J_F=(x_2-x_1)^2) shows that the leader settles at x₁ = 1 while the follower tracks the best‑response x₂ = 2, confirming SE convergence. In the canonical “Fish War” game, the Stackelberg nES yields a leader‑favored allocation, contrasting with the symmetric NE obtained by the standard nES configuration.

The key contribution is the insight that equilibrium selection (NE versus SE) can be controlled purely by the hierarchical scaling of the algorithm’s parameters, without altering the underlying closed‑loop dynamics. This opens a pathway for model‑free hierarchical optimization in multi‑time‑scale engineering systems such as smart power grids, networked resource allocation, and particle‑accelerator tuning, where a leader‑follower structure naturally arises. The paper also enriches the theoretical toolbox by combining Lie‑bracket averaging with singular‑perturbation analysis to handle nested, time‑scale‑separated, model‑free optimization schemes.