Decentralized Reinforcement Learning for Multi-Agent Multi-Resource Allocation via Dynamic Cluster Agreements

This paper addresses the challenge of allocating heterogeneous resources among multiple agents in a decentralized manner. Our proposed method, Liquid-Graph-Time Clustering-IPPO, builds upon Independent Proximal Policy Optimization (IPPO) by integrating dynamic cluster consensus, a mechanism that allows agents to form and adapt local sub-teams based on resource demands. This decentralized coordination strategy reduces reliance on global information and enhances scalability. We evaluate LGTC-IPPO against standard multi-agent reinforcement learning baselines and a centralized expert solution across a range of team sizes and resource distributions. Experimental results demonstrate that LGTC-IPPO achieves more stable rewards, better coordination, and robust performance even as the number of agents or resource types increases. Additionally, we illustrate how dynamic clustering enables agents to reallocate resources efficiently also for scenarios with discharging resources.

💡 Research Summary

**
The paper tackles the challenging problem of allocating heterogeneous resources among multiple agents in a fully decentralized fashion. The authors propose a novel algorithm called Liquid‑Graph‑Time Clustering‑Independent Proximal Policy Optimization (LGTC‑IPPO), which builds on Independent PPO (IPPO) but augments it with a dynamic cluster consensus mechanism. In this framework, agents continuously form and dissolve sub‑teams (clusters) based on the current demand profile of consumers. Within each cluster, agents share a consensus value function, thereby addressing the credit‑assignment problem without requiring a global value estimator.

The problem is formalized as a Dec‑POMDP: N agents operate in a bounded continuous space, delivering r different resource types to M consumers. Resources are either persistent (requiring agents to stay in the interaction area) or instantaneous (delivered upon arrival). Agents receive only local observations (their own position, held resources, and the positions and demands of all consumers) and can communicate only with neighbors within a radius C. The objective is to maximize a discounted sum of individual rewards, which are carefully designed to balance global and local objectives.

Reward Design
The total reward for each agent is a sum of several components:

• Global demand reduction (rw_g) – proportional to the total decrease in unmet demand across all consumers.
• Coverage reward (rw_s) – a binary bonus if at least one agent is present at every consumer location.
• Collision penalty (rw_ij) – negative term proportional to the squared distance between agents that come closer than a safety threshold.
• Instantaneous resource delivery reward (rw_im) – positive reward for delivering an instantaneous resource.
• Persistent resource reward (rw_is) – fixed reward for satisfying a persistent demand.
• Cluster completion reward (rw_rc) – bonus for a sub‑team that fully satisfies a consumer’s demand.
• Assignment‑driven positional reward (rw_id) – derived from solving a mixed‑integer quadratic program (MIQP) that computes an optimal binary assignment matrix a; agents receive a reward proportional to the reduction in distance to their assigned consumer.

These components are weighted empirically to ensure that agents are incentivized both to cooperate globally and to focus on the specific consumer they are assigned to.

Neural Architecture and Dynamic Consensus
Two identical neural networks are used for policy and value estimation. Each consumer’s resource information is processed by a DeepSets module (Φ_m), while each agent’s own state is encoded by another DeepSets module (Ψ_i). The consumer feature matrix Φ̅ and agent feature matrix Ψ̅ are combined through a graph filter defined by the adjacency (or Laplacian) matrix S and a series of learnable weight tensors (B_k, A_k, etc.). The resulting combined input Δ is passed through a softmax‑scaled attention mechanism (Ξ) that assigns importance scores to each consumer for every agent. The attention scores are then used to select the dominant consumer features influencing the agents’ hidden state x.

The dynamics of the hidden state are described by a neural ordinary differential equation (ODE) (Equation 6). Under two mild assumptions—bounded bias terms and uniformly bounded graph norms—the authors prove (Theorem 1) that the ODE is infinitesimally contractive, guaranteeing that the hidden state remains within a bounded interval