Digital-Twin Empowered Deep Reinforcement Learning For Site-Specific Radio Resource Management in NextG Wireless Aerial Corridor

Joint base station (BS) association and beam selection in multi-UAV aerial corridors constitutes a challenging radio resource management (RRM) problem. It is driven by high-dimensional action spaces, need for substantial overhead to acquire global channel state information (CSI), rapidly varying propagation channels, and stringent latency requirements. Conventional combinatorial optimization methods, while near-optimal, are computationally prohibitive for real-time operation in such dynamic environments. While learning-based approaches can mitigate computational complexity and CSI overhead, the need for extensive site-specific (SS) datasets for model training remains a key challenge. To address these challenges, we develop a Digital Twin (DT)-enabled two-stage optimization framework that couples physics-based beam gain modeling with DRL for scalable online decision-making. In the first stage, a channel twin (CT) is constructed using a high-fidelity ray-tracing solver with geo-spatial contexts, and network information to capture SS propagation characteristics, and dual annealing algorithm is employed to precompute optimal transmission beam directions. In the second stage, a Multi-Head Proximal Policy Optimization (MH-PPO) agent, equipped with a scalable multi-head actor-critic architecture, is trained on the DT-generated channel dataset to directly map complex channel and beam states to jointly execute UAV-BS-beam association decisions. The proposed PPO agent achieves a 44%-121% improvement over DQN and 249%-807% gain over traditional heuristic based optimization schemes in a dense UAV scenario, while reducing inference latency by several orders of magnitude. These results demonstrate that DT-driven training pipelines can deliver high-performance, low-latency RRM policies tailored to SS deployments suitable for real-time resource management in next-generation aerial corridor networks.

💡 Research Summary

This paper tackles the joint UAV‑base‑station (BS) association and beam selection problem that arises in next‑generation (NextG) aerial corridor networks, where multiple unmanned aerial vehicles (UAVs) share a limited set of terrestrial BSs equipped with directional antenna arrays. The authors argue that conventional combinatorial optimization, while near‑optimal, is computationally infeasible for real‑time operation because it requires global channel state information (CSI) and iterative search, both of which are costly in highly dynamic aerial environments. To overcome these limitations, they propose a two‑stage digital‑twin (DT) enabled framework that couples a physics‑based channel twin (CT) with a deep reinforcement learning (DRL) agent.

Stage 1 – Channel Twin Construction.
A high‑fidelity CT is built using NVIDIA’s Sionna simulator, which integrates a ray‑tracing engine (Mitsuba‑3) with TensorFlow. The authors model a realistic 3‑D campus environment (Howard University) by importing OpenStreetMap building footprints into Blender, assigning ITU material profiles (concrete, marble, metal), and placing four terrestrial BSs using OpenCelliD coordinates. Each BS carries a 4 × 4 uniform planar array operating at 3.5 GHz with a 3GPP codebook of orthogonal beams. UAV positions are sampled uniformly within a 3‑D corridor at altitudes of 60 m, 80 m, or 100 m, producing a diverse set of line‑of‑sight (LOS) and non‑LOS conditions. For every BS‑UAV‑beam link, the ray‑tracer emits 10⁶ rays with up to five interactions, yielding a complex‑valued channel impulse response (CIR) tensor Z∈ℂ^{M×L×N×K}. The channel gain matrix H is obtained by aggregating the power of all multipath components, and mean angles of arrival (θ̄, ϕ̄) are computed for use in the antenna gain model (3GPP TR 37.840). This CT provides a site‑specific, physics‑consistent dataset that captures the true propagation characteristics of the environment.

Stage 2 – DRL Policy Learning.
Using the CT‑generated dataset, the authors train a Multi‑Head Proximal Policy Optimization (MH‑PPO) agent. The problem is formalized as a binary matching: each UAV must be assigned to exactly one BS‑beam pair, each beam can serve at most one UAV, and the objective is to maximize the network sum‑capacity while mitigating inter‑cell and intra‑cell interference. Because the problem is NP‑hard, it is reformulated as a policy‑learning task. The MH‑PPO architecture consists of a shared trunk that encodes the global state (CSI, UAV coordinates, beam indices) and a set of dedicated actor heads, one per UAV, that output discrete actions (selected BS and beam). This design scales linearly with the number of UAVs and avoids the combinatorial explosion of a monolithic action space.

A dual‑annealing algorithm is employed offline to pre‑compute optimal transmission directions for each BS‑beam pair, providing a physically meaningful initialization for the DRL agent and reducing exploration overhead. The reward function directly reflects the instantaneous sum‑rate; additionally, a penalty term is introduced when more than N UAVs select the same BS, thereby encouraging load‑balanced associations without hard constraints. The PPO’s clipped surrogate objective ensures stable policy updates, while the multi‑head setup allows independent learning per UAV while still sharing global information through the trunk.

Simulation Setup and Results.
The authors evaluate the framework in a realistic 3‑D scenario with four BSs, fourteen buildings, and varying UAV densities (5–30 UAVs). Channel parameters follow ECC Report 281 and 3GPP TR 37.840. Baselines include (i) a heuristic strongest‑signal association, (ii) a dual‑annealing based combinatorial optimizer, and (iii) a Deep Q‑Network (DQN) DRL agent. The MH‑PPO policy achieves a 44 %–121 % throughput gain over DQN and a 249 %–807 % gain over the heuristic, while inference latency drops from hundreds of milliseconds (optimizers) to sub‑millisecond levels, satisfying stringent real‑time requirements. The policy also exhibits robust generalization across different UAV altitudes and densities, and the multi‑head architecture maintains low memory footprint (≈200 MB) and linear inference time growth with UAV count.

Contributions and Significance.

1. DT‑driven training pipeline: A physics‑consistent CT is constructed and used to generate a massive site‑specific dataset, eliminating the need for costly field measurements.
2. MH‑PPO framework: A scalable multi‑head actor‑critic design that directly maps high‑dimensional CSI to joint UAV‑BS‑beam decisions, handling interference and load balancing through learned behavior.
3. Performance and latency: Demonstrated orders‑of‑magnitude latency reduction and substantial throughput improvements, confirming suitability for real‑time NextG aerial corridor operations.

Limitations and Future Work.
The CT generation requires high‑performance GPUs and considerable offline computation, and the current study assumes static UAV positions; extending to dynamic trajectories and online DT updates is an open challenge. Future directions include (i) online DT synchronization for real‑time channel updates, (ii) transfer learning to adapt policies to new sites with minimal retraining, (iii) integration with massive MIMO or mmWave beamforming, and (iv) joint optimization of energy consumption and cooperative transmission among UAVs.

In summary, the paper presents a novel integration of high‑fidelity digital twins and multi‑head proximal policy optimization to solve the joint UAV‑BS‑beam association problem efficiently and scalably, offering a practical pathway toward site‑specific, low‑latency radio resource management in next‑generation aerial corridor networks.