Generalization in Reinforcement Learning for Radio Access Networks

Modern RAN operate in highly dynamic and heterogeneous environments, where hand-tuned, rule-based RRM algorithms often underperform. While RL can surpass such heuristics in constrained settings, the diversity of deployments and unpredictable radio conditions introduce major generalization challenges. Data-driven policies frequently overfit to training conditions, degrading performance in unseen scenarios. To address this, we propose a generalization-centered RL framework for RAN control that: (i) robustly reconstructs dynamically varying states from partial and noisy observations, while encoding static and semi-static information, such as radio nodes, cell attributes, and their topology, through graph representations; (ii) applies domain randomization to broaden the training distribution; and (iii) distributes data generation across multiple actors while centralizing training in a cloud-compatible architecture aligned with O-RAN principles. Although generalization increases computational and data-management complexity, our distributed design mitigates this by scaling data collection and training across diverse network conditions. Applied to downlink link adaptation in five 5G benchmarks, our policy improves average throughput and spectral efficiency by ~10% over an OLLA baseline (10% BLER target) in full-buffer MIMO/mMIMO and by >20% under high mobility. It matches specialized RL in full-buffer traffic and achieves up to 4- and 2-fold gains in eMBB and mixed-traffic benchmarks, respectively. In nine-cell deployments, GAT models offer 30% higher throughput over MLP baselines. These results, combined with our scalable architecture, offer a path toward AI-native 6G RAN using a single, generalizable RL agent.

💡 Research Summary

The paper tackles a fundamental obstacle to deploying reinforcement‑learning (RL) based radio resource management (RRM) in modern 5G/6G radio access networks (RAN): the lack of generalization. Conventional rule‑based algorithms struggle in highly dynamic, heterogeneous environments, and RL policies trained on a fixed set of conditions tend to overfit, leading to severe performance drops when faced with unseen cells, traffic patterns, or channel conditions. To overcome this, the authors propose a generalization‑centric RL framework that integrates three complementary enablers: (1) robust state reconstruction via graph‑based representations, (2) extensive domain randomization to diversify the training distribution, and (3) a scalable distributed learning architecture aligned with O‑RAN principles.

State reconstruction: The framework treats each network element (cells, antennas, users) as nodes in a graph, encoding static attributes (cell ID, antenna count, bandwidth) and semi‑static information (topology, neighbor relations) as node features, while dynamic measurements (SINR, CQI, buffer occupancy) are attached as time‑varying attributes. A Graph Attention Network (GAT) processes this heterogeneous graph, allowing the model to focus on the most relevant neighbors and to fuse static and dynamic information into a compact latent state. This approach mitigates partial observability and noisy measurements, which are typical in real‑world RAN deployments.

Domain randomization: During training, the authors systematically randomize a wide range of environment parameters—transmit power, user density, mobility speed, channel models (including fading and line‑of‑sight variations), traffic loads, and scheduling policies. By exposing the agent to a broad “sim‑to‑sim” distribution, the learned policy acquires robustness to variations that would otherwise cause over‑specialization. The technique is especially effective for high‑mobility scenarios, where the policy retains performance despite rapid channel fluctuations.

Distributed learning architecture: To generate the massive, diverse experience required for the above randomization, the authors deploy many parallel actors (simulators) each running a different randomization seed and network topology. Experiences are streamed to a central cloud server that maintains a replay buffer enriched with environment metadata. The central learner samples from this buffer to update the policy using a proximal policy optimization (PPO) variant. This design respects O‑RAN’s xApp/rApp model, enables horizontal scaling of both data collection and compute, and provides a unified model that can be deployed network‑wide.

The framework is evaluated on a high‑fidelity 5G system‑level simulator across five benchmark scenarios: full‑buffer MIMO, massive MIMO (mMIMO), high‑mobility, eMBB, and mixed‑traffic. The RL task is downlink link adaptation (LA), where the agent selects modulation‑coding schemes and transmission power to meet a 10 % BLER target while maximizing throughput. Results show:

• Average cell throughput and spectral efficiency improve by ~10 % compared with the state‑of‑the‑art outer‑loop link adaptation (OLLA) baseline in full‑buffer MIMO/mMIMO cases.
• High‑mobility (>120 km/h) sees >20 % gain, confirming the benefit of domain randomization.
• In eMBB and mixed‑traffic benchmarks, the generalized policy achieves up to 4× and 2× throughput gains respectively, outperforming specialized RL agents that were tuned for each scenario.
• When scaling to a nine‑cell deployment, the GAT‑based policy delivers 30 % higher throughput than a multi‑layer perceptron (MLP) baseline, highlighting the advantage of graph‑structured representations in larger topologies.

The authors also analyze the reward structure, demonstrating that the policy maintains the target BLER while reducing unnecessary transmit power, indicating efficient use of radio resources. They acknowledge increased computational and data‑management overhead due to graph processing and distributed training, and note that a gap remains between simulation and live network conditions, necessitating further field trials.

In conclusion, the paper presents a concrete pathway toward an AI‑native 6G RAN where a single, generalizable RL agent can replace multiple hand‑crafted heuristics across diverse cells and traffic conditions. Future work is suggested on model compression, meta‑RL for rapid adaptation, and real‑world validation, aiming to bridge the remaining simulation‑to‑deployment gap.