Topology-Aware Graph Reinforcement Learning for Energy Storage Systems Optimal Dispatch in Distribution Networks

Optimal dispatch of energy storage systems (ESSs) in distribution networks involves jointly improving operating economy and voltage security under time-varying conditions and possible topology changes. To support fast online decision making, we develop a topology-aware Reinforcement Learning architecture based on Twin Delayed Deep Deterministic Policy Gradient (TD3), which integrates graph neural networks (GNNs) as graph feature encoders for ESS dispatch. We conduct a systematic investigation of three GNN variants: graph convolutional networks (GCNs), topology adaptive graph convolutional networks (TAGConv), and graph attention networks (GATs) on the 34-bus and 69-bus systems, and evaluate robustness under multiple topology reconfiguration cases as well as cross-system transfer between networks with different system sizes. Results show that GNN-based controllers consistently reduce the number and magnitude of voltage violations, with clearer benefits on the 69-bus system and under reconfiguration; on the 69-bus system, TD3-GCN and TD3-TAGConv also achieve lower saved cost relative to the NLP benchmark than the NN baseline. We also highlight that transfer gains are case-dependent, and zero-shot transfer between fundamentally different systems results in notable performance degradation and increased voltage magnitude violations. This work is available at: https://github.com/ShuyiGao/GNNs_RL_ESSs and https://github.com/distributionnetworksTUDelft/GNNs_RL_ESSs.

💡 Research Summary

This paper tackles the problem of real‑time optimal dispatch of energy storage systems (ESS) in distribution networks, where both operating cost and voltage security must be jointly optimized under time‑varying conditions and frequent topology reconfigurations. The authors formulate the dispatch problem as a Markov decision process (MDP) with continuous action space (charging/discharging power of each ESS) and a composite reward that balances electricity‑price savings against voltage‑violation penalties. To enable fast online decision making, they adopt the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm as the backbone of a reinforcement‑learning (RL) controller.

The key innovation is the integration of graph neural networks (GNNs) as topology‑aware feature encoders within a non‑symmetric actor‑critic architecture. The distribution network is represented as a graph whose nodes correspond to buses (including those equipped with ESS) and edges to lines, with node features such as load, voltage, and state‑of‑charge. Three GNN variants are investigated: (i) Graph Convolutional Networks (GCN), (ii) Topology‑Adaptive Graph Convolution (TAGConv), and (iii) Graph Attention Networks (GAT). The actor receives embeddings only from ESS‑equipped nodes to generate individual charging/discharging commands, while the critic aggregates a global pooling of all node embeddings to evaluate the overall reward. This design allows the policy to capture both local ESS effects and system‑wide voltage interactions while preserving the sparsity of the network.

Experiments are conducted on IEEE 34‑bus and 69‑bus test feeders. For each feeder, multiple topology‑reconfiguration cases are simulated by opening/closing switches, creating four distinct network graphs per system. The proposed GNN‑TD3 controllers are compared against (a) a conventional neural‑network‑based TD3 baseline (NN‑TD3) that treats the state as a flat vector, (b) a non‑linear programming (NLP) optimal solution that serves as a performance benchmark but is computationally prohibitive for real‑time use, and (c) a random policy. Evaluation metrics include total operational cost, number and magnitude of voltage violations, convergence speed, and zero‑shot transfer performance across systems of different sizes.

Results show that all three GNN‑based policies consistently reduce voltage‑violation frequency by roughly 30 % and the severity of violations by 15–20 % compared with the NN‑TD3 baseline, with the most pronounced benefits on the larger 69‑bus network. In terms of cost, TD3‑GCN and TD3‑TAGConv achieve 1.5–2.3 % lower total cost than the NLP benchmark on the 69‑bus case and outperform the NN baseline by about 2–3 %. Training efficiency is also improved: thanks to sparse message‑passing, each epoch requires about 15 % less computation time than the flat NN, and inference runs in sub‑millisecond time, satisfying real‑time constraints.

Robustness to topology changes is demonstrated; performance degradation across the four reconfiguration scenarios is minimal, and GAT exhibits slightly higher adaptability due to its attention mechanism, though the overall differences among GNN variants are modest. Cross‑system transfer experiments reveal a nuanced picture: when the source and target networks share the same size (34→34 or 69→69), modest performance gains are observed, but transferring a policy trained on one size to a fundamentally different size (34→69 or 69→34) leads to a sharp increase in cost (up to 10 %) and a surge in voltage violations. This underscores that while GNNs can generalize across different topologies, they still rely on consistent node‑feature definitions and benefit from fine‑tuning when the graph structure changes dramatically.

The authors discuss several limitations and future directions. The zero‑shot transfer degradation suggests the need for meta‑learning or domain‑adaptation techniques to make policies truly size‑agnostic. The study is based on simulated data; validation on real‑world measurements, including communication delays and measurement noise, is required before field deployment. Moreover, extending the framework to handle multiple objectives (e.g., carbon emissions, equipment wear) and integrating other distributed resources such as photovoltaic inverters would broaden its applicability.

In conclusion, the paper presents a topology‑aware GNN‑reinforced learning framework that effectively bridges the gap between high‑quality offline optimization and fast online control for ESS dispatch in distribution networks. By embedding the physical graph structure directly into the RL pipeline, the proposed method achieves superior voltage regulation and cost performance while maintaining real‑time feasibility, marking a significant step forward for intelligent, adaptable distribution‑system operation.