A Constrained RL Approach for Cost-Efficient Delivery of Latency-Sensitive Applications

Next-generation networks aim to provide performance guarantees to real-time interactive services that require timely and cost-efficient packet delivery. In this context, the goal is to reliably deliver packets with strict deadlines imposed by the application while minimizing overall resource allocation cost. A large body of work has leveraged stochastic optimization techniques to design efficient dynamic routing and scheduling solutions under average delay constraints; however, these methods fall short when faced with strict per-packet delay requirements. We formulate the minimum-cost delay-constrained network control problem as a constrained Markov decision process and utilize constrained deep reinforcement learning (CDRL) techniques to effectively minimize total resource allocation cost while maintaining timely throughput above a target reliability level. Results indicate that the proposed CDRL-based solution can ensure timely packet delivery even when existing baselines fall short, and it achieves lower cost compared to other throughput-maximizing methods.

💡 Research Summary

The paper tackles a pressing challenge in next‑generation (NextG) networks: delivering packets for real‑time interactive (RTI) services within strict per‑packet deadlines while minimizing the operational cost of network resources. Existing approaches, such as Back‑Pressure, Universal Max‑Weight (UMW), and Universal Cloud Network Control (UCNC), are designed for average‑delay constraints and therefore cannot guarantee that each packet meets a hard deadline (time‑to‑live, TTL). When packets miss their TTL they become useless, rendering traditional queue‑stability conditions ineffective for cost‑efficient control.

To address this gap, the authors formulate the Minimum‑Cost Delay‑Constrained Network Control (MDNC) problem as a Constrained Markov Decision Process (CMDP). The network is modeled as a directed graph G = (V, E). Each link (i, j) can allocate up to Xmaxij resource blocks; each block provides a capacity Cbij packets per slot and incurs a cost eij (e.g., power). Multiple latency‑sensitive commodities c ∈ C are defined, each with a source sc, destination dc, initial lifetime Lc, and a reliability target δc (the required fraction of packets delivered on time). Packet arrivals at sources are stochastic with mean rate \bar b_c.

The system state at time t comprises, for every node i, the vector of incoming packet arrivals and the TTL‑segmented queue backlogs q_{c,ℓ}^i(t). The action consists of three components: (i) resource‑block allocation x_{ij}(t), (ii) routing and scheduling flows f_{c,ℓ}^{ij}(t) (the number of packets of commodity c with remaining lifetime ℓ sent over link (i, j)), and (iii) proactive dropping decisions g_{c,ℓ}^i(t). The dynamics follow a lifetime‑based queueing law: packets age by one TTL unit each slot, expire when ℓ = 0, and are removed upon reaching their destination with positive remaining lifetime.

The objective is to minimize the long‑run average cost
 lim_{T→∞} (1/T) Σ_{t=0}^{T−1} Σ_{(i,j)∈E} E