EgoHandICL: Egocentric 3D Hand Reconstruction with In-Context Learning

Robust 3D hand reconstruction in egocentric vision is challenging due to depth ambiguity, self-occlusion, and complex hand-object interactions. Prior methods mitigate these issues by scaling training data or adding auxiliary cues, but they often struggle in unseen contexts. We present EgoHandICL, the first in-context learning (ICL) framework for 3D hand reconstruction that improves semantic alignment, visual consistency, and robustness under challenging egocentric conditions. EgoHandICL introduces complementary exemplar retrieval guided by vision-language models (VLMs), an ICL-tailored tokenizer for multimodal context, and a masked autoencoder (MAE)-based architecture trained with hand-guided geometric and perceptual objectives. Experiments on ARCTIC and EgoExo4D show consistent gains over state-of-the-art methods. We also demonstrate real-world generalization and improve EgoVLM hand-object interaction reasoning by using reconstructed hands as visual prompts. Code and data: https://github.com/Nicous20/EgoHandICL

💡 Research Summary

EgoHandICL introduces the first in‑context learning (ICL) framework for egocentric 3D hand reconstruction from monocular RGB images. The authors identify three core challenges in egocentric settings—depth ambiguity, self‑occlusion, and complex hand‑object interactions—and argue that human reasoning, which leverages prior experience and multimodal cues, aligns naturally with the ICL paradigm. The proposed system consists of three tightly coupled components: (1) exemplar retrieval, (2) a multimodal ICL tokenizer, and (3) a masked‑autoencoder (MAE)‑style reconstruction network.

Exemplar Retrieval. To supply the ICL model with relevant demonstrations, EgoHandICL employs a vision‑language model (VLM) in two complementary ways. First, a set of predefined visual templates classifies each image into one of four hand‑involvement types (left‑hand, right‑hand, both‑hands, no‑hand) and retrieves a visually consistent exemplar from a database. Second, an adaptive textual template mechanism prompts the VLM with user‑specified natural‑language descriptions (e.g., “the scissors occlude the right hand”) and retrieves a semantically aligned exemplar based on text similarity. By combining visual and textual cues, the method ensures that the selected template shares both appearance and semantic context with the query image.

Multimodal ICL Tokenizer. For each query‑template pair, four token streams are generated: (i) image tokens from a pretrained Vision Transformer (ViT), (ii) structural tokens obtained by feeding both coarse MANO parameters (produced by an off‑the‑shelf hand reconstructor such as HaMeR or WiLoR) and ground‑truth MANO parameters into a MANO encoder, (iii) text tokens derived from the VLM‑generated description, and (iv) target tokens representing the ground‑truth MANO parameters to be predicted. Cross‑attention layers fuse these streams into a unified ICL token sequence, effectively bridging the 2D‑3D modality gap.

MAE‑Style Learning and Hand‑Guided Losses. The reconstruction network follows a masked‑autoencoder design: a random subset of target tokens from both template and query is masked during training, and the transformer learns to reconstruct them from the unmasked context. At inference time, all query target tokens are masked, forcing the model to predict the full set of MANO parameters solely from the multimodal context. The training objective combines (a) a geometric L2 loss on MANO parameters, (b) a mesh‑vertex L2 loss after decoding the MANO model, and (c) perceptual losses based on rendered silhouettes and depth maps, encouraging visual consistency.

Experiments. EgoHandICL is evaluated on two egocentric benchmarks, ARCTIC and EgoExo4D. It consistently outperforms state‑of‑the‑art methods (HaMeR, WiLoR, etc.) by 7‑9 % reduction in mean per‑joint position error (MPJPE), with pronounced gains in severely occluded scenarios, two‑hand crossings, and cases involving dark gloves. Qualitative results demonstrate that the retrieved exemplars provide crucial semantic hints that resolve ambiguous depth cues. The authors also test the system on self‑captured egocentric videos and integrate the reconstructed hand meshes as visual prompts for an EgoVLM model. This integration improves hand‑object interaction reasoning, confirming the utility of the reconstructed geometry beyond pure pose estimation.

Limitations and Future Work. Currently only a single exemplar is used per query, and the retrieval relies heavily on the VLM’s embedding quality. Scaling to multiple exemplars, improving retrieval efficiency, and extending the hand‑guided losses to handle complex lighting or reflective surfaces are identified as future directions. The authors suggest incorporating additional sensors (e.g., depth or light‑field cameras) and exploring dynamic exemplar composition to further enhance robustness.

Conclusion. EgoHandICL demonstrates that in‑context learning, when equipped with VLM‑driven exemplar selection and a multimodal tokenization scheme, can substantially improve egocentric 3D hand reconstruction. By unifying visual, textual, and structural information within a masked‑autoencoder framework, the method achieves both geometric accuracy and visual fidelity, offering a practical solution for XR, HCI, and robotics applications that require real‑time, high‑precision hand modeling in challenging first‑person viewpoints.