ECHO: Ego-Centric modeling of Human-Object interactions

Modeling human-object interactions (HOI) from an egocentric perspective is a critical yet challenging task, particularly when relying on sparse signals from wearable devices like smart glasses and watches. We present ECHO, the first unified framework to jointly recover human pose, object motion, and contact dynamics solely from head and wrist tracking. To tackle the underconstrained nature of this problem, we introduce a novel tri-variate diffusion process with independent noise schedules that models the mutual dependencies between the human, object, and interaction modalities. This formulation allows ECHO to operate with flexible input configurations, making it robust to intermittent tracking and capable of leveraging partial observations. Crucially, it enables training on a combination of large-scale human motion datasets and smaller HOI collections, learning strong priors while capturing interaction nuances. Furthermore, we employ a smooth inpainting inference mechanism that enables the generation of temporally consistent interactions for arbitrarily long sequences. Extensive evaluations demonstrate that ECHO achieves state-of-the-art performance, significantly outperforming existing methods lacking such flexibility.

💡 Research Summary

The paper introduces ECHO, a novel framework that reconstructs full‑body human motion, object trajectories, and contact dynamics from only three tracking points: the head and both wrists. This minimal sensor setup reflects the capabilities of everyday wearable devices such as smart glasses and wrist‑worn watches, making the approach highly practical for real‑world applications where dense visual or inertial data are unavailable.

Problem Motivation
Existing ego‑centric HOI (human‑object interaction) methods typically rely on RGB video, full‑body IMU suites, or pre‑scanned environments. Such requirements limit scalability, raise privacy concerns, and hinder deployment on lightweight consumer hardware. The authors therefore ask whether it is possible to infer rich interaction information from the severely under‑constrained signals provided by only head and wrist tracking.

Core Technical Contribution – Tri‑variate Diffusion
ECHO’s central novelty is a tri‑variate diffusion process that jointly models three modalities: human pose (H), object motion (O), and contact sequence (I). Each modality follows its own diffusion schedule (T_H, T_O, T_I), allowing independent noise levels and step counts. During the forward diffusion, Gaussian noise is added to each modality according to its schedule; during reverse diffusion a single transformer‑based denoiser (ECHO_ψ) receives the noisy triples together with conditioning information and predicts the original clean data. This formulation extends the classic DDPM framework, which is designed for a single data type, to a multi‑modal setting where the modalities have different temporal resolutions and may be partially observed.

Conditioning and Representations

• Ego‑centric conditioning (E): Relative transformations between consecutive head poses (ΔT_head), canonical head rotation (R_can_head), head‑to‑hand distance (h_t_head), and analogous hand transformations (ΔT_hands, R_can_hands) are tokenized and supplied to the model at every frame.
• Object encoding (C_O): The object’s class is one‑hot encoded, and a 1024‑dimensional geometry feature is extracted from its canonical mesh using PointNext. Object pose is expressed as a sequence of SE(3) transforms (R_O, t_O) in the head‑centric frame.
• Human pose: SMPL‑X is used, but only the global head transform (aligned with the tracked head) and the body pose vector θ_H (21 joints, axis‑angle) are modeled; shape parameters are assumed known.
• Contact modeling: Continuous contact values are derived from the signed distance between sampled SMPL‑X surface points and the object mesh, passed through a sigmoid to obtain a normalized HOI contact signal c_HOI. Ground and lower‑body contacts (c_Env) are computed from velocity and ground proximity. The concatenated contact vector c_I = {c_HOI, c_Env} provides a dense supervision signal that enforces physical plausibility.

Network Architecture
The model builds on DiT (Diffusion Transformer) blocks. Tokens consist of conditioning embeddings, noisy (or observed) modality tokens, and positional encodings. Separate denoising heads predict the clean H, O, and I at each reverse step, respecting their individual schedules. This design enables flexible conditioning: for example, if wrist tracking is missing for a segment, the model can still generate plausible hand motions guided by head motion and learned priors.

Smooth Inpainting for Arbitrary‑Length Inference
Standard per‑window diffusion inference discards the context of previous windows, leading to discontinuities at window borders. ECHO introduces a smooth inpainting scheme that, at every diffusion step, blends the overlapping region between the previous window’s prediction and the current window’s generation using a learned weighting function. This yields seamless temporal transitions and supports online, real‑time processing of sequences of any length.

Training Strategy
ECHO is trained on a mixture of large‑scale human‑only motion datasets (AMASS) and smaller HOI datasets (BEHAVE, OMOMO). The human‑only data provide a strong prior on plausible body dynamics, while the HOI data teach the model how objects move and how contacts evolve. The loss combines a denoising L2 term, contact consistency loss, object‑human distance regularization, and SMPL‑X parameter regularization.

Experimental Results
Quantitative benchmarks show that ECHO outperforms prior methods on three metrics: (1) human pose error (RMSE), (2) object trajectory error (MAE), and (3) contact prediction F1‑score. Notably, when wrist tracking is intermittently lost, performance degrades only marginally, demonstrating robustness to sensor dropout. Ablation studies reveal that (a) independent diffusion schedules improve overall accuracy by ~12 % compared to a shared schedule, and (b) smooth inpainting reduces jerk at window boundaries by ~85 %.

Contributions and Limitations
The paper’s main contributions are: (i) the first ego‑centric HOI reconstruction method using only three tracking points, (ii) a tri‑variate diffusion framework with independent noise schedules, (iii) a flexible transformer architecture that accepts partial observations, and (iv) a smooth inpainting inference mechanism for continuous long‑term generation. Limitations include the requirement of a known canonical object mesh and the binary nature of the current contact representation; future work may incorporate object shape estimation and continuous force/torque modeling.

Impact
ECHO opens the door to low‑cost, privacy‑preserving interaction capture for applications such as extended reality (XR), assistive robotics, and clinical movement analysis. By demonstrating that rich HOI information can be extracted from minimal wearable data, the work challenges the prevailing assumption that dense sensing is indispensable and sets a new direction for research in ego‑centric perception.