Efficient UAV trajectory prediction: A multi-modal deep diffusion framework

To meet the requirements for managing unauthorized UAVs in the low-altitude economy, a multi-modal UAV trajectory prediction method based on the fusion of LiDAR and millimeter-wave radar information is proposed. A deep fusion network for multi-modal UAV trajectory prediction, termed the Multi-Modal Deep Fusion Framework, is designed. The overall architecture consists of two modality-specific feature extraction networks and a bidirectional cross-attention fusion module, aiming to fully exploit the complementary information of LiDAR and radar point clouds in spatial geometric structure and dynamic reflection characteristics. In the feature extraction stage, the model employs independent but structurally identical feature encoders for LiDAR and radar. After feature extraction, the model enters the Bidirectional Cross-Attention Mechanism stage to achieve information complementarity and semantic alignment between the two modalities. To verify the effectiveness of the proposed model, the MMAUD dataset used in the CVPR 2024 UG2+ UAV Tracking and Pose-Estimation Challenge is adopted as the training and testing dataset. Experimental results show that the proposed multi-modal fusion model significantly improves trajectory prediction accuracy, achieving a 40% improvement compared to the baseline model. In addition, ablation experiments are conducted to demonstrate the effectiveness of different loss functions and post-processing strategies in improving model performance. The proposed model can effectively utilize multi-modal data and provides an efficient solution for unauthorized UAV trajectory prediction in the low-altitude economy.

💡 Research Summary

The paper addresses the pressing need for accurate, real‑time prediction of unauthorized unmanned aerial vehicle (UAV) trajectories in the emerging low‑altitude economy. Recognizing that single‑sensor approaches (vision, LiDAR, or radar alone) suffer from environmental constraints, the authors propose a multimodal deep fusion framework that jointly exploits LiDAR point clouds and millimeter‑wave radar returns. The architecture consists of two parallel, structurally identical feature encoders based on PointNet, each equipped with a channel‑attention module to weight the most informative feature channels. After independent encoding, a bidirectional cross‑attention mechanism aligns and enriches the modalities: LiDAR features serve as queries while radar features act as keys and values, and the reverse direction is processed symmetrically. This design enables each sensor to compensate for the other’s weaknesses—LiDAR provides high‑resolution spatial geometry, while radar supplies robust velocity and reflectivity cues under adverse weather or occlusion.

The fused representation is obtained by element‑wise addition of the original modality features and the two cross‑attention‑enhanced features, followed by two fully‑connected layers with ReLU activation and dropout, yielding a 3‑D position estimate. For regression, the authors adopt Smooth L1 loss, which behaves like L2 for small errors (ensuring fast convergence) and like L1 for large errors (reducing sensitivity to outliers). An ablation comparing RMSE loss versus Smooth L1 shows a dramatic reduction in position RMSE from 3.20 m to 1.78 m, confirming the robustness of the chosen loss.

Post‑processing is essential because the network processes single frames without temporal context, leading to occasional jitter or spikes caused by sensor noise or synchronization errors. The authors introduce two complementary strategies: (1) outlier detection that flags any frame whose displacement exceeds a 2 m threshold, and (2) a sliding‑window average (window size = 5) that smooths the trajectory. Using only outlier detection reduces position RMSE to 1.61 m but inflates speed error; combining both yields the best overall performance (position RMSE = 1.67 m, speed RMSE = 1.38 m/s).

Experiments are conducted on the MMAUD dataset, released for the CVPR 2024 UG2+ UAV Tracking and Pose‑Estimation Challenge. The dataset provides synchronized LiDAR, radar, stereo vision, and audio streams; the study uses only LiDAR and radar. Training data comprise UAV models Mavic 2, Mavic 3, and Phame, while M300 serves as the unseen test set, demonstrating cross‑model generalization. Temporal alignment between modalities is achieved by nearest‑timestamp matching and zero‑padding to maintain consistent tensor shapes.

Compared against a baseline that employs single‑modality LiDAR with a Kalman filter, the multimodal bidirectional attention model achieves a 40 % improvement in both position (2.79 m → 1.67 m) and velocity (1.73 m/s → 1.38 m/s) errors. The paper’s contributions are threefold: (1) an efficient dual‑encoder with channel attention for high‑quality feature extraction, (2) a bidirectional cross‑attention fusion that aligns heterogeneous sensor data at the semantic level, and (3) a robust training and post‑processing pipeline (Smooth L1 loss + outlier detection + smoothing) that yields stable, accurate trajectory predictions.

Limitations include the current reliance on frame‑wise processing without explicit temporal modeling; the authors suggest future integration of recurrent or transformer modules to capture motion dynamics. Additionally, the approach depends on well‑annotated training data and has not been validated under extreme interference or dense clutter scenarios. Prospective work may incorporate additional modalities (RGB cameras, microphone arrays) and explore self‑supervised or spatio‑temporal graph networks to further enhance robustness and generalization across diverse operational environments.