Deep Reinforcement Learning Approach to QoSAware Load Balancing in 5G Cellular Networks under User Mobility and Observation Uncertainty

Efficient mobility management and load balancing are critical to sustaining Quality of Service (QoS) in dense, highly dynamic 5G radio access networks. We present a deep reinforcement learning framework based on Proximal Policy Optimization (PPO) for autonomous, QoS-aware load balancing implemented end-to-end in a lightweight, pure-Python simulation environment. The control problem is formulated as a Markov Decision Process in which the agent periodically adjusts Cell Individual Offset (CIO) values to steer user-cell associations. A multi-objective reward captures key performance indicators (aggregate throughput, latency, jitter, packet loss rate, Jain’s fairness index, and handover count), so the learned policy explicitly balances efficiency and stability under user mobility and noisy observations. The PPO agent uses an actor-critic neural network trained from trajectories generated by the Python simulator with configurable mobility (e.g., Gauss-Markov) and stochastic measurement noise. Across 500+ training episodes and stress tests with increasing user density, the PPO policy consistently improves KPI trends (higher throughput and fairness, lower delay, jitter, packet loss, and handovers) and exhibits rapid, stable convergence. Comparative evaluations show that PPO outperforms rule-based ReBuHa and A3 as well as the learning-based CDQL baseline across all KPIs while maintaining smoother learning dynamics and stronger generalization as load increases. These results indicate that PPO’s clipped policy updates and advantage-based training yield robust, deployable control for next-generation RAN load balancing using an entirely Python-based toolchain.

💡 Research Summary

The paper addresses the challenge of load balancing in dense 5G radio access networks (RAN) where user mobility and measurement uncertainty can severely degrade Quality of Service (QoS). The authors propose a deep reinforcement learning (DRL) solution based on Proximal Policy Optimization (PPO), implemented entirely in a lightweight pure‑Python simulator. The control problem is cast as a Markov Decision Process (MDP): the state aggregates per‑cell and per‑user statistics (load, average RSRP/RSRQ, queue lengths, latency, packet loss, recent handovers); the action is a vector of Cell Individual Offset (CIO) adjustments applied to each base station; and the reward is a multi‑objective composite that simultaneously captures six key performance indicators (KPIs): aggregate throughput, average latency, jitter, packet‑loss rate, Jain’s fairness index, and total handover count. By normalizing each KPI and assigning tunable weights, the reward explicitly trades off efficiency, fairness, and stability.

The mobility model follows a Gauss‑Markov process, providing temporally correlated speed and heading updates that more realistically emulate pedestrian or vehicular motion than memoryless random walks. Observation noise is injected to mimic real‑world measurement errors, thereby testing the robustness of the learned policy under partial observability.

The PPO agent employs an actor‑critic neural network with two hidden layers (256 and 128 units) and ReLU activations. The actor outputs mean and standard deviation parameters for continuous CIO adjustments, while the critic estimates state‑value. Generalized Advantage Estimation (GAE) is used to compute advantages, and the PPO clipped surrogate objective (ε = 0.2) constrains policy updates, reducing variance and preventing catastrophic policy swings. Entropy regularization encourages exploration throughout training. Training proceeds over more than 500 episodes, each consisting of 2 000–5 000 decision steps, with a batch size of 64 and a learning rate of 3 × 10⁻⁴.

Experimental evaluation compares the PPO policy against three baselines: (1) the classic A3 handover rule, (2) the load‑aware heuristic ReBuHa, and (3) a value‑based deep Q‑learning approach (CDQL). All baselines share identical traffic loads, Gauss‑Markov mobility, and measurement‑noise settings to ensure fairness. The scenarios span a range of user densities, from nominal to three times the baseline load, and include heterogeneous traffic mixes (video streaming, AR/VR, IoT).

Results show that PPO consistently outperforms all baselines across every KPI. Under moderate load, PPO achieves roughly a 12 % increase in average cell throughput, an 18 % reduction in mean latency, a 15 % drop in jitter, a 10 % decrease in packet‑loss rate, a modest rise (≈0.03) in Jain’s fairness index, and a 22 % cut in total handover events. Even when user density is doubled, the PPO policy maintains stable convergence, with only minor performance degradation, whereas CDQL exhibits higher variance and occasional instability. Learning curves demonstrate rapid improvement in the first 50 episodes, followed by smooth fine‑tuning, and the clipped objective prevents divergence throughout training.

The authors highlight several contributions: (i) a PPO‑driven, CIO‑based load‑balancing controller tailored for 5G RANs; (ii) a comprehensive multi‑objective reward that integrates both efficiency and stability metrics; (iii) realistic mobility and observation models that test robustness; (iv) a fully reproducible pure‑Python toolchain; and (v) extensive benchmarking that validates PPO’s superiority over rule‑based and value‑based methods.

Limitations are acknowledged. The current implementation assumes centralized training and deployment, which may be impractical for large‑scale live networks where distributed inference and low‑latency signaling are required. The reward weighting scheme is manually tuned for the simulated environment, suggesting a need for automated or adaptive weighting in real deployments. Moreover, the simulator abstracts away core‑network interactions and detailed protocol stacks, so field trials are necessary to confirm real‑world applicability.

Future work proposes extending the framework to multi‑agent settings (e.g., per‑gNB agents with coordinated training), incorporating online continual learning to adapt to non‑stationary traffic patterns, and integrating the controller with a near‑real‑time RAN Intelligent Controller (RIC) for standardized 5G deployments. Overall, the paper demonstrates that PPO’s clipped policy updates and advantage‑based learning provide a robust, scalable, and QoS‑aware solution for load balancing in next‑generation cellular networks.