HandMCM: Multi-modal Point Cloud-based Correspondence State Space Model for 3D Hand Pose Estimation

3D hand pose estimation that involves accurate estimation of 3D human hand keypoint locations is crucial for many human-computer interaction applications such as augmented reality. However, this task poses significant challenges due to self-occlusion of the hands and occlusions caused by interactions with objects. In this paper, we propose HandMCM to address these challenges. Our HandMCM is a novel method based on the powerful state space model (Mamba). By incorporating modules for local information injection/filtering and correspondence modeling, the proposed correspondence Mamba effectively learns the highly dynamic kinematic topology of keypoints across various occlusion scenarios. Moreover, by integrating multi-modal image features, we enhance the robustness and representational capacity of the input, leading to more accurate hand pose estimation. Empirical evaluations on three benchmark datasets demonstrate that our model significantly outperforms current state-of-the-art methods, particularly in challenging scenarios involving severe occlusions. These results highlight the potential of our approach to advance the accuracy and reliability of 3D hand pose estimation in practical applications.

💡 Research Summary

The paper introduces HandMCM, a novel framework for 3D hand pose estimation that tackles the long‑standing challenges of self‑occlusion and hand‑object occlusion. HandMCM takes as input an RGB‑D pair and a set of 3D points sampled from the depth map. A multi‑modal super‑point encoder first extracts local 3D geometry using a PointNet‑style layer and visual features from RGB and depth images via separate ResNet‑based autoencoders. These 2D features are projected into 3D space and concatenated with the point cloud to form a dense, multi‑modal super‑point representation. From this representation, keypoint tokens are generated using a bias‑induced layer that provides per‑joint embeddings.

The core contribution is the Correspondence Mamba block, which leverages the Mamba state‑space model (SSM) to model dynamic kinematic relationships among hand joints. Tokens are normalized and split into forward and backward sequences, each processed by a bidirectional gated SSM (BiGS). The forward and backward hidden states are combined via an outer‑product to produce a dynamic correspondence map that captures both directions of interaction. This map multiplies the transformed token vector, yielding an updated token that encodes global kinematic context. A final linear projection maps these high‑dimensional tokens to 3D joint coordinates.

To compensate for the loss of local geometric detail, the authors add a local token injection and filtering mechanism. For each joint, the k‑nearest super‑points and their features are aggregated with the joint token using a SetConv operation, injecting precise local geometry into the Mamba block. This hybrid of global dynamic correspondence and local geometry improves robustness, especially for heavily occluded joints.

Extensive experiments on three benchmarks—NYU, DexYCB, and HO3D—show that HandMCM achieves state‑of‑the‑art performance, reporting mean joint errors of 7.06 mm (NYU), 6.67 mm (DexYCB), and 1.71 cm (HO3D). Ablation studies confirm that each component (multi‑modal super‑point encoding, Correspondence Mamba, and local token injection) contributes significantly to the overall accuracy. The authors also release their code, facilitating reproducibility.

In summary, HandMCM is the first hand‑pose estimator that directly applies a modern SSM architecture to capture dynamic kinematic correspondences while integrating rich multi‑modal visual cues and fine‑grained local geometry. Its superior performance under severe occlusion makes it a promising candidate for real‑time applications in AR/VR, robotics, and human‑computer interaction.