Multi-Agent Pointer Transformer: Seq-to-Seq Reinforcement Learning for Multi-Vehicle Dynamic Pickup-Delivery Problems

This paper addresses the cooperative Multi-Vehicle Dynamic Pickup and Delivery Problem with Stochastic Requests (MVDPDPSR) and proposes an end-to-end centralized decision-making framework based on sequence-to-sequence, named Multi-Agent Pointer Transformer (MAPT). MVDPDPSR is an extension of the vehicle routing problem and a spatio-temporal system optimization problem, widely applied in scenarios such as on-demand delivery. Classical operations research methods face bottlenecks in computational complexity and time efficiency when handling large-scale dynamic problems. Although existing reinforcement learning methods have achieved some progress, they still encounter several challenges: 1) Independent decoding across multiple vehicles fails to model joint action distributions; 2) The feature extraction network struggles to capture inter-entity relationships; 3) The joint action space is exponentially large. To address these issues, we designed the MAPT framework, which employs a Transformer Encoder to extract entity representations, combines a Transformer Decoder with a Pointer Network to generate joint action sequences in an AutoRegressive manner, and introduces a Relation-Aware Attention module to capture inter-entity relationships. Additionally, we guide the model’s decision-making using informative priors to facilitate effective exploration. Experiments on 8 datasets demonstrate that MAPT significantly outperforms existing baseline methods in terms of performance and exhibits substantial computational time advantages compared to classical operations research methods.

💡 Research Summary

The paper introduces the Multi-Agent Pointer Transformer (MAPT), a novel end-to-end centralized decision-making framework designed to solve the Multi-Vehicle Dynamic Pickup and Delivery Problem with Stochastic Requests (MVDPDPSR). This problem, which is critical in on-demand delivery logistics, involves optimizing routes for multiple vehicles as new, unpredictable requests emerge in real-time. While classical Operations Research (OR) methods struggle with computational complexity in large-scale dynamic scenarios, existing Reinforcement Learning (RL) approaches face difficulties in modeling joint action distributions, capturing inter-entity relationships, and managing the exponentially expanding action space.

To overcome these challenges, the authors propose the MAPT framework, which integrates a Transformer-based architecture with specialized modules for multi-agent coordination. The first major innovation is the use of an AutoRegressive decoding process via a Transformer Decoder and a Pointer Network. Unlike previous methods that sample actions for each vehicle independently—often leading to conflicts such as duplicate task assignments—MAPT models the selection of vehicles and requests as a single, sequential process. This ensures that the joint action distribution is captured, preventing inefficient or overlapping assignments.

The second innovation is the Relation-Aware Attention module. Traditional encoders often fail to capture the structural dependencies between vehicles, requests, and stops. MAPT addresses this by incorporating a relation matrix $R$ into the Scalable Dot-Product Attention. By embedding distance-based relationships and learnable parameters into the attention scores, the model can inherently learn semantic connections, such as the proximity of stops and the availability of certain requests.

Thirdly, to mitigate the explosion of the joint action space, the authors introduce “Informative Priors.” By designing priors based on load balancing and stop distances, the model is guided toward more promising regions of the action space during exploration. These priors are fused with the decoder’s output probability distribution, significantly reducing the randomness of the exploration phase and accelerating the convergence of the REINFORCE-based policy gradient training.

Experimental results across eight diverse datasets demonstrate the superiority of MAPT. The proposed model achieved a 12% to 18% improvement in total value compared to state-of-the-art baselines, including metaheuristics like Tabu Search and advanced RL models like MAPPO and Attention-VRP. Furthermore, MAPT demonstrated a significant computational advantage, performing decisions more than five times faster than traditional OR methods. Ablation studies further confirm that both the Relation-Aware Attention and the Informative Priors are essential components for achieving high-performance, real-time routing in complex, dynamic environments. This makes MAPT a highly practical and scalable solution for modern, large-scale logistics optimization.