Gaussian Belief Propagation Network for Depth Completion

Depth completion aims to predict a dense depth map from a color image with sparse depth measurements. Although deep learning methods have achieved state-of-the-art (SOTA), effectively handling the sparse and irregular nature of input depth data in deep networks remains a significant challenge, often limiting performance, especially under high sparsity. To overcome this limitation, we introduce the Gaussian Belief Propagation Network (GBPN), a novel hybrid framework synergistically integrating deep learning with probabilistic graphical models for end-to-end depth completion. Specifically, a scene-specific Markov Random Field (MRF) is dynamically constructed by the Graphical Model Construction Network (GMCN), and then inferred via Gaussian Belief Propagation (GBP) to yield the dense depth distribution. Crucially, the GMCN learns to construct not only the data-dependent potentials of MRF but also its structure by predicting adaptive non-local edges, enabling the capture of complex, long-range spatial dependencies. Furthermore, we enhance GBP with a serial & parallel message passing scheme, designed for effective information propagation, particularly from sparse measurements. Extensive experiments demonstrate that GBPN achieves SOTA performance on the NYUv2 and KITTI benchmarks. Evaluations across varying sparsity levels, sparsity patterns, and datasets highlight GBPN’s superior performance, notable robustness, and generalizable capability.

💡 Research Summary

Depth completion, the task of generating a dense depth map from a color image together with sparse depth measurements, has seen rapid progress thanks to deep learning. However, existing neural‑network‑only approaches still struggle with the irregular, highly sparse nature of the input depth, leading to degraded performance and poor robustness when measurements are scarce. In this paper the authors propose a novel hybrid framework called the Gaussian Belief Propagation Network (GBPN) that unifies deep representation learning with probabilistic graphical inference.

The core idea is to treat dense depth as a set of random variables defined on the image lattice and to model their joint distribution with a Markov Random Field (MRF). Unlike traditional MRF‑based methods that use hand‑crafted smoothness terms and a fixed graph structure, GBPN learns a scene‑specific MRF end‑to‑end. A dedicated Graphical Model Construction Network (GMCN) takes the RGB image (and optionally an intermediate depth estimate) and predicts:

1. Unary potentials – confidence weights (w_i) for each measured pixel and the measured depth value (s_i).
2. Pairwise potentials – edge weights (w_{ij}) and expected depth differences (r_{ij}) for every edge.
3. Dynamic non‑local edges – a set of long‑range connections whose existence and strength are inferred from image content, allowing the model to capture relationships far beyond the usual 8‑connected neighbourhood.

Because the MRF is Gaussian, inference can be performed with Gaussian Belief Propagation (GBP). In GBP each belief and each message is represented by a mean (\mu) and a precision (\Lambda) (or canonical parameters (\eta)). The authors derive closed‑form update equations for both belief aggregation and message passing, which reduce to simple linear algebraic operations even for high‑resolution images.

A key contribution is the serial‑and‑parallel message passing scheme. Early iterations propagate messages serially, ensuring that the strong signal from the sparse depth measurements quickly reaches nearby pixels. After a few rounds, the algorithm switches to parallel updates, allowing the whole graph to be refined simultaneously. This hybrid schedule dramatically improves information flow from the sparse measurements to distant, unmeasured regions. To stabilize loopy belief propagation, the authors employ damping (a weighted average of the current and previous messages) and graph decomposition tricks, achieving convergence within 10–15 iterations in practice.

Training is performed end‑to‑end with a probability‑based loss that penalizes both the deviation of the predicted mean from ground‑truth depth and the uncertainty (precision) of each pixel. This encourages the network not only to output accurate depth values but also reliable confidence estimates, which are valuable for downstream risk‑aware tasks such as autonomous navigation or robotic manipulation.

The method is evaluated on the two most widely used depth‑completion benchmarks: NYUv2 (indoor) and KITTI (outdoor). GBPN consistently outperforms previous state‑of‑the‑art methods, including CSPN‑based refiners and recent transformer‑based regressors, achieving lower RMSE and MAE across all sparsity levels. Notably, when the input depth is extremely sparse (1–5 % of pixels), GBPN’s performance degrades only marginally, whereas competing methods suffer a sharp drop. Ablation studies confirm that each component is essential: removing the dynamic graph (using a fixed 8‑connected grid) leads to a large accuracy loss; fixing the potentials instead of updating them during inference slows convergence; and using only a parallel message schedule reduces the propagation of sparse cues, harming performance under high sparsity.

The authors acknowledge limitations: the Gaussian assumption may be insufficient for highly non‑linear depth phenomena such as specular reflections or transparent surfaces, and the dynamic graph construction adds computational overhead that could be problematic for real‑time robotics. Future work is suggested in the direction of non‑Gaussian belief propagation, variational inference, or tighter integration with graph neural networks to further improve expressiveness and efficiency.

In summary, GBPN demonstrates a powerful synergy between deep learning and structured probabilistic inference. By learning a scene‑adaptive MRF and solving it with an efficient, hybrid GBP scheme, the framework naturally handles irregular sparse inputs, provides global consistency, and yields per‑pixel uncertainty. This represents a significant step forward for depth completion and opens new possibilities for robust 3D perception in autonomous systems.