CaFe-TeleVision: A Coarse-to-Fine Teleoperation System with Immersive Situated Visualization for Enhanced Ergonomics

Teleoperation presents a promising paradigm for remote control and robot proprioceptive data collection. Despite recent progress, current teleoperation systems still suffer from limitations in efficiency and ergonomics, particularly in challenging scenarios. In this paper, we propose CaFe-TeleVision, a coarse-to-fine teleoperation system with immersive situated visualization for enhanced ergonomics. At its core, a coarse-to-fine control mechanism is proposed in the retargeting module to bridge workspace disparities, jointly optimizing efficiency and physical ergonomics. To stream immersive feedback with adequate visual cues for human vision systems, an on-demand situated visualization technique is integrated in the perception module, which reduces the cognitive load for multi-view processing. The system is built on a humanoid collaborative robot and validated with six challenging bimanual manipulation tasks. User study among 24 participants confirms that CaFe-TeleVision enhances ergonomics with statistical significance, indicating a lower task load and a higher user acceptance during teleoperation. Quantitative results also validate the superior performance of our system across six tasks, surpassing comparative methods by up to 28.89% in success rate and accelerating by 26.81% in completion time. Project webpage: https://clover-cuhk.github.io/cafe_television/

💡 Research Summary

CaFe‑TeleVision introduces a novel teleoperation framework that simultaneously tackles two persistent challenges in remote robot control: (1) the mismatch between human operator workspace and robot task space, which hampers efficiency and causes physical strain, and (2) the cognitive overload caused by conventional multi‑view visual feedback, which forces frequent gaze shifts and introduces occlusions. The system combines a coarse‑to‑fine retargeting mechanism with an on‑demand situated visualization technique, aiming to improve both operational efficiency and ergonomics.

The retargeting module operates in two complementary modes. In the “coarse” (natural) mode, the operator’s wrist pose is scaled and aligned to respect the robot’s joint limits, thereby preserving physical ergonomics and reducing muscular fatigue during large‑scale motions. When finer adjustments are required, the system seamlessly switches to a “fine” (joystick‑assisted) mode, where a joystick provides high‑resolution control of end‑effector pose. This dual‑mode architecture allows users to stay in the flow of the task while dynamically balancing speed and precision without any pause in control.

The perception module addresses visual ergonomics by streaming stereoscopic video from an eye‑mounted ZED 2i camera as the primary display and overlaying a gripper‑anchored view only on demand. This “situated” visualization presents data in spatial proximity to the object of interest, preserving contextual cues while eliminating unnecessary visual clutter. Two wrist‑mounted RealSense D435i cameras supply additional dynamic cues, ensuring that motion parallax and object deformation are observable in real time. By reducing eye‑focus shifts, visual distraction, and occlusion, the system markedly lowers cognitive load.

Hardware integration is built around a Franka Emika Panda collaborative robot. Human motion is captured at 60 Hz using Xsens IMU and VR controllers; robot joint commands and gripper actuation run at 60 Hz, eye‑camera streams at 15 Hz, and wrist cameras at 30 Hz. All streams are processed on a GPU‑accelerated Unity application, delivering 1080p stereoscopic video at 15 Hz to a VR head‑mounted display via Air Link.

The authors evaluated the platform on six challenging bimanual tasks (e.g., fruit picking, cap twisting, tea pouring, towel hanging, bag packing) with 24 participants. Three conditions were compared: (a) traditional natural mode, (b) joystick‑assisted mode, and (c) the proposed CaFe‑TeleVision. Metrics included success rate, task completion time, NASA‑TLX workload, and SUS user acceptance. CaFe‑TeleVision achieved up to a 28.89 % increase in success rate, a 26.81 % reduction in completion time, an average 18 % decrease in perceived workload, and statistically higher user acceptance scores.

The study demonstrates that a coarse‑to‑fine retargeting strategy can resolve workspace disparities while preserving physical ergonomics, and that on‑demand situated visualization can substantially improve cognitive ergonomics. Together, these contributions redefine the efficiency‑ergonomics trade‑off in human‑robot teleoperation, offering a scalable solution applicable to domains such as medical surgery, space exploration, and hazardous environment manipulation. Future work will explore broader robot platforms, adaptive mode‑switching policies, and integration with higher‑level autonomy to further enhance system robustness and usability.