Mean-Field Control on Sparse Graphs: From Local Limits to GNNs via Neighborhood Distributions

Mean-field control (MFC) offers a scalable solution to the curse of dimensionality in multi-agent systems but traditionally hinges on the restrictive assumption of exchangeability via dense, all-to-all interactions. In this work, we bridge the gap to real-world network structures by proposing a rigorous framework for MFC on large sparse graphs. We redefine the system state as a probability measure over decorated rooted neighborhoods, effectively capturing local heterogeneity. Our central contribution is a theoretical foundation for scalable reinforcement learning in this setting. We prove horizon-dependent locality: for finite-horizon problems, an agent’s optimal policy at time t depends strictly on its (T-t)-hop neighborhood. This result renders the infinite-dimensional control problem tractable and underpins a novel Dynamic Programming Principle (DPP) on the lifted space of neighborhood distributions. Furthermore, we formally and experimentally justify the use of Graph Neural Networks (GNNs) for actor-critic algorithms in this context. Our framework naturally recovers classical MFC as a degenerate case while enabling efficient, theoretically grounded control on complex sparse topologies.

💡 Research Summary

**
This paper addresses a fundamental limitation of classical mean‑field control (MFC): the assumption that agents interact through a complete, all‑to‑all graph, which guarantees exchangeability and allows the system state to be represented by a single probability measure over agent states. Real‑world multi‑agent systems—social networks, power grids, biological swarms, robotic fleets—are typically sparse, with each agent only observing a small, possibly heterogeneous neighborhood. In such settings the empirical state distribution is no longer a sufficient statistic, and the classical MFC framework collapses.

The authors propose a rigorous “graph‑lifted” MFC framework that replaces the traditional mean‑field by a probability measure over decorated rooted graphs. A decorated rooted graph consists of a rooted subgraph together with a “decoration” that assigns a state from the agent state space to each vertex. The lifted state µ_t ∈ P(G_X^*) thus captures both the local topology and the local states of agents. This construction is motivated by the theory of local weak convergence (Benjamini–Schramm convergence): for a sequence of sparse random graphs (e.g., Erdős–Rényi with average degree d) the k‑hop neighborhoods of a uniformly chosen vertex converge in distribution to the k‑hop neighborhood of a random rooted limit graph. Consequently, as the number of agents N → ∞, the empirical distribution of decorated neighborhoods converges to a deterministic measure µ_t that evolves according to a Markovian transition operator T_t.

The central theoretical contribution is the Horizon‑Dependent Locality Theorem (Theorem 2). For a finite horizon T, the optimal policy at time t depends only on the (T − t)‑hop neighborhood of the agent. The proof combines three ingredients: (i) the dynamics and reward functions are assumed to be local, i.e., they depend on a bounded radius k; (ii) the lifted state µ_t evolves deterministically under T_t, preserving the Markov property; (iii) a dynamic programming principle (DPP) is established on the lifted space, showing that the Bellman recursion can be restricted to measures supported on neighborhoods of decreasing radius as time progresses. This result dramatically reduces the infinite‑dimensional control problem to a sequence of finite‑dimensional subproblems, each involving only local information.

Building on the locality result, the authors formulate a Dynamic Programming Principle on P(G_X^*). They define a value function V_t(µ) and prove the existence of an optimal stationary policy under standard continuity and compactness assumptions. The Bellman equation takes the same form as in classical MFC but now operates on the space of neighborhood distributions.

From an algorithmic perspective, the locality theorem suggests a natural architecture: Graph Neural Networks (GNNs). Since the optimal policy at time t aggregates information from exactly (T − t) hops, a GNN that repeatedly applies message‑passing for (T − t) layers can represent the optimal mapping. The paper proposes a GNN‑based actor‑critic algorithm. The actor network receives as input the agent’s local decorated subgraph, processes it through a depth‑equal‑to‑remaining‑horizon GNN, and outputs a distribution over actions. The critic network shares the same GNN backbone to estimate the local value function. The authors provide a policy‑gradient theorem adapted to the lifted setting and a propagation‑of‑chaos style result that guarantees that the mean‑field limit approximates the finite‑N system as N grows.

Empirical evaluation is performed on several sparse graph families: Erdős–Rényi graphs with varying average degree, small‑world networks, and a real‑world power‑grid topology. Baselines include classical MF‑Q‑learning, global‑mean‑field actor‑critic, and heuristic local‑average methods. Across all experiments, the GNN‑actor‑critic converges faster and achieves higher cumulative rewards, especially for longer horizons where deeper GNNs can exploit the additional locality information. Ablation studies confirm that respecting the (T − t)‑hop radius is crucial: shallow GNNs (insufficient depth) underperform, while overly deep GNNs (exceeding the required radius) do not yield further gains.

The framework also subsumes classical MFC as a degenerate case: when the interaction graph is complete or when rewards and dynamics depend only on the root’s state, the decorated rooted graph distribution collapses to the ordinary state distribution µ ∈ P(X). Thus the proposed theory is a genuine extension rather than a replacement.

In conclusion, the paper delivers a comprehensive theory for mean‑field control on large sparse graphs, establishing a precise locality property, a dynamic programming formulation on lifted neighborhood distributions, and a principled justification for GNN‑based reinforcement learning. It bridges the gap between the elegant but restrictive classical MFC and the messy reality of sparse networked multi‑agent systems. Future directions suggested include handling dynamic or time‑varying graphs, partial observability and communication constraints, and extending the analysis to continuous‑time dynamics and continuous state spaces.