Foundation Model-Aided Hierarchical Deep Reinforcement Learning for Blockage-Aware Link in RIS-Assisted Networks

Reconfigurable intelligent surface (RIS) technology has the potential to significantly enhance the spectral efficiency (SE) of 6G wireless networks. However, practical deployment remains constrained by challenges in accurate channel estimation and control optimization under dynamic conditions. This paper presents a foundation model-aided hierarchical deep reinforcement learning (FM-HDRL) framework designed for joint beamforming and phase-shift optimization in RIS-assisted wireless networks. To implement this, we first fine-tune a pre-trained large wireless model (LWM) to translate raw channel data into low-dimensional, context-aware channel state information (CSI) embeddings. Next, these embeddings are combined with user location information and blockage status to select the optimal communication path. The resulting features are then fed into an HDRL model, assumed to be implemented at a centralized controller, which jointly optimizes the base station (BS) beamforming vectors and the RIS phase-shift configurations to maximize SE. Simulation results demonstrate that the proposed FM-HDRL framework consistently outperforms baseline methods in terms of convergence speed, spectral efficiency, and scalability. According to the simulation results, our proposed method improves 7.82% SE compared to the FM-aided deep reinforcement learning (FM-DRL) approach and a substantial enhancement of about 48.66% relative to the beam sweeping approach.

💡 Research Summary

The paper tackles the challenge of improving spectral efficiency (SE) in blockage‑prone millimeter‑wave (mmWave) 6G networks by leveraging reconfigurable intelligent surfaces (RIS). Traditional RIS‑assisted solutions suffer from high overhead in channel state information (CSI) acquisition and the computational burden of jointly optimizing base‑station (BS) beamforming and RIS phase‑shifts, especially under dynamic user mobility and blockage conditions. To address these issues, the authors propose a novel framework called FM‑HDRL (Foundation Model‑aided Hierarchical Deep Reinforcement Learning).

The first component of FM‑HDRL is a large wireless model (LWM), a transformer‑based foundation model pre‑trained on massive wireless channel datasets. By fine‑tuning only the final layer of LWM on a RIS‑specific dataset, the model learns to compress raw high‑dimensional channel matrices into low‑dimensional context‑aware embeddings (denoted E). This process reduces pilot‑based estimation overhead while preserving essential propagation characteristics such as multipath components and the interaction among BS, RIS, and users.

The second component is a two‑level hierarchical deep reinforcement learning (HDRL) architecture modeled as a semi‑Markov decision process (SMDP). The high‑level meta‑controller operates on a slower timescale (macro‑slot) and observes the spatial distribution of users together with binary blockage indicators bₖ. It decides, for each user, whether to use the direct line‑of‑sight (LoS) link or the RIS‑assisted non‑LoS path, forming a binary goal vector gₕₜ. The meta‑controller’s reward is the cumulative sum‑SE obtained by the low‑level controller during the macro‑slot, penalized when any user’s SE falls below a predefined minimum R_min, thereby enforcing fairness.

The low‑level sub‑controller works at each time slot within the macro‑slot. Its state consists of the LWM‑generated CSI embedding E and the high‑level goal bₜ. It selects continuous actions: the BS precoding matrix W and the RIS phase‑shift matrix Θ. The immediate reward is the instantaneous sum‑SE across all users. Both controllers are implemented using Deep Deterministic Policy Gradient (DDPG) actors and critics, each with two hidden layers of 256 units, learning rates of 1e‑4, and a discount factor γ = 0.99. Experience replay buffers (size 500 for the meta‑controller, 400 for the sub‑controller) enable off‑policy updates.

Simulation settings include a BS with N = 32 antennas, an RIS with M = 32 passive elements, and K = 50 mobile single‑antenna users moving randomly. For each user, the 10 strongest propagation paths are generated, and a frequency‑flat channel model is assumed. The LWM processes channel matrices split into 32 patches, each embedded into a 128‑dimensional space, and the fine‑tuned model projects these to a lower‑dimensional embedding. Training of LWM uses a batch size of 64, learning rate 1e‑5, and mean‑squared‑error loss.

Performance results show that FM‑HDRL outperforms two baselines: (1) FM‑DRL, which uses the same foundation model but a single‑level DRL agent, and (2) a conventional beam‑sweeping approach. FM‑HDRL achieves a 7.82 % increase in average SE over FM‑DRL and a 48.66 % gain over beam sweeping. Moreover, FM‑HDRL converges significantly faster and scales gracefully as the number of RIS elements grows, whereas FM‑DRL experiences instability due to the exploding action‑space dimensionality. The hierarchical decomposition aligns the control timescales with the physical dynamics (slow blockage events vs. fast fading), leading to more stable learning and better exploitation of the CSI embeddings.

In conclusion, the paper demonstrates that (i) large‑scale foundation models can provide compact, high‑quality CSI representations, (ii) hierarchical reinforcement learning can effectively separate long‑term strategic decisions (link selection) from short‑term resource allocation (beamforming and phase‑shifts), and (iii) the combined FM‑HDRL framework yields tangible SE improvements and faster convergence in realistic RIS‑assisted networks. The authors suggest future work on distributed meta‑controllers, multi‑RIS coordination, and real‑time hardware validation to further bridge the gap between simulation and deployment.