Multi-agent Assessment with QoS Enhancement for HD Map Updates in a Vehicular Network

Reinforcement Learning (RL) algorithms have been used to address the challenging problems in the offloading process of vehicular ad hoc networks (VANET). More recently, they have been utilized to improve the dissemination of high-definition (HD) Maps. Nevertheless, implementing solutions such as deep Q-learning (DQN) and Actor-critic at the autonomous vehicle (AV) may lead to an increase in the computational load, causing a heavy burden on the computational devices and higher costs. Moreover, their implementation might raise compatibility issues between technologies due to the required modifications to the standards. Therefore, in this paper, we assess the scalability of an application utilizing a Q-learning single-agent solution in a distributed multi-agent environment. This application improves the network performance by taking advantage of a smaller state, and action space whilst using a multi-agent approach. The proposed solution is extensively evaluated with different test cases involving reward function considering individual or overall network performance, number of agents, and centralized and distributed learning comparison. The experimental results demonstrate that the time latencies of our proposed solution conducted in voice, video, HD Map, and best-effort cases have significant improvements, with 40.4%, 36%, 43%, and 12% respectively, compared to the performances with the single-agent approach.

💡 Research Summary

The paper addresses the challenge of delivering high‑definition (HD) map updates over vehicular ad‑hoc networks (VANETs) while meeting stringent quality‑of‑service (QoS) requirements for multiple concurrent services (voice, video, best‑effort, and HD map). Existing approaches that embed deep reinforcement learning (e.g., DQN, A3C) directly on autonomous vehicles suffer from high computational load, increased hardware cost, and often require modifications to the IEEE 802.11p MAC layer, limiting practical deployment.

To overcome these limitations, the authors extend a previously proposed single‑agent Q‑learning algorithm into a lightweight multi‑agent framework. Each vehicle (or each service class) acts as an independent agent that selects a contention‑window (CW) value for its transmissions. All agents share the same reward function:

U(c) = α₁·R(c)/R_max – α₂·L(c)/L_max + F,

where R(c) and L(c) are the throughput and latency for service class c, α₁ and α₂ balance the trade‑off, and F adds penalties/bonuses to improve stability. By using an identical reward across agents, the system conveys network‑wide performance information without explicit state or action sharing, thereby avoiding extra control‑plane traffic.

Two learning architectures are examined: (1) centralized learning, where agents periodically synchronize a common Q‑table via a central server, and (2) fully distributed learning, where each agent updates its own Q‑values locally. Simulations reveal that the distributed version consistently yields lower average latency—by 5–8%—especially under high vehicle density, because it eliminates synchronization delays and reduces channel contention.

The experimental setup models IEEE 802.11p V2V communication with realistic mobility patterns and varying traffic mixes. Results show substantial latency reductions compared with the original single‑agent approach: voice latency improves by 40.4 %, video by 36 %, HD‑map by 43 %, and best‑effort by 12 %. Moreover, the Q‑learning agents require roughly 30 %–45 % less computational effort than deep‑RL counterparts, making them suitable for on‑board units with modest processing capabilities.

Key contributions include:

1. A novel distributed multi‑agent Q‑learning scheme that reduces state‑action dimensionality, lowers computational complexity, and operates entirely at the application layer, preserving compatibility with existing IEEE 802.11p standards.
2. Two distinct multi‑agent configurations (service‑based and vehicle‑based agents) demonstrating flexibility in resource allocation strategies.
3. Comprehensive evaluation of centralized vs. distributed learning, providing insights into trade‑offs between convergence speed and real‑time performance.
4. Integration of heterogeneous services into a single optimization framework, a gap in prior work that often considered only a single traffic type.

In summary, the study proves that a lightweight, reward‑consistent multi‑agent Q‑learning approach can effectively manage contention windows in VANETs, delivering significant QoS gains for HD map dissemination and other latency‑sensitive services while remaining computationally feasible for current vehicle hardware and without requiring changes to the MAC protocol. This work paves the way for scalable, standards‑compliant V2X solutions that can accommodate the growing diversity of vehicular applications.