A Modern System Recipe for Situated Embodied Human-Robot Conversation with Real-Time Multimodal LLMs and Tool-Calling

Situated embodied conversation requires robots to interleave real-time dialogue with active perception: deciding what to look at, when to look, and what to say under tight latency constraints. We present a simple, minimal system recipe that pairs a real-time multimodal language model with a small set of tool interfaces for attention and active perception. We study six home-style scenarios that require frequent attention shifts and increasing perceptual scope. Across four system variants, we evaluate turn-level tool-decision correctness against human annotations and collect subjective ratings of interaction quality. Results indicate that real-time multimodal large language models and tool use for active perception is a promising direction for practical situated embodied conversation.

💡 Research Summary

The paper presents a compact yet powerful recipe for enabling situated embodied human‑robot conversation by tightly coupling a real‑time multimodal large language model (LLM) with a small set of tool‑calling interfaces that control attention and active perception. The authors argue that simply plugging a real‑time LLM into a robot is insufficient; the robot must also decide where and when to look, acquire visual evidence on demand, and maintain shared attention during dialogue. To address this, they pair state‑of‑the‑art streaming LLMs (OpenAI Realtime and Google Gemini Live) with five carefully designed functions: look_at_person, look_at_object, look_around, look_for, and use_vision.

The system architecture consists of two core components. The first is the real‑time multimodal LLM, which continuously consumes microphone audio and egocentric video, handles voice‑activity detection, turn‑taking, and incremental speech generation. The second component is the tool‑calling layer that exposes robot capabilities as JSON‑schema functions. During a conversation turn, the LLM may emit a function call with typed arguments; the robot executes the call synchronously, updates its perceptual context, and feeds any new visual information back to the LLM. This design offloads the traditionally hand‑engineered decision logic for gaze control to the LLM’s learned priors, while still guaranteeing low‑latency execution through lightweight on‑device perception modules.

Implementation details are noteworthy. Person tracking uses YOLO‑pose (≈20 Hz) and object mask estimation uses SAM (≈6 Hz), both running locally to keep the gaze‑control loop at frame‑rate speeds. look_at_person and look_at_object continuously compute the required yaw/pitch adjustments from keypoints or mask centroids and publish the target pose to the robot controller. look_around performs a scripted sweep over a list of predefined target coordinates, captures an image at each pose, and stores the tuple (image, robot pose, field‑of‑view) in a lightweight “view memory”. look_for takes a natural‑language query, scores all stored views with a vision‑language model (VLM) in parallel, selects the highest‑scoring view, re‑orients the robot to that pose, and sends the image and query back to the LLM. use_vision allows the LLM to request the latest frame only when visual grounding is needed, reducing token overhead.

The authors evaluate the approach across six home‑style scenarios that progressively increase perceptual scope: posture coaching, whiteboard tutoring, lamp placement, plant diagnosis, outfit checking, and misplaced‑item finding. For each scenario they collect turn‑level human annotations of “what the robot should do next” and compare them against the tool calls generated by four system variants: (1) OpenAI vs. Gemini backend, (2) with vs. without tool calling, (3) with vs. without the view‑memory map, and (4) with vs. without the VLM‑based look_for module. Objective metrics measure tool‑call correctness; subjective metrics (questionnaires) assess fluency, social presence, and perceived situatedness.

Results show that the full configuration (real‑time LLM + tool calling + view memory + VLM‑based look_for) achieves the highest correctness (~85 %) and the best subjective scores (average 4.3/5). The inclusion of look_for and use_vision is especially beneficial in tasks requiring out‑of‑view search (e.g., finding a misplaced item), where the robot can autonomously sweep the environment, retrieve the most relevant stored view, and seamlessly integrate that visual evidence into the ongoing dialogue. Ablations that remove tool calling or the view memory lead to noticeable drops in both objective and subjective performance, confirming that the tool layer is not a mere convenience but a critical component for real‑time situated interaction.

The paper’s contributions are threefold: (1) a clear system architecture that pairs a streaming multimodal LLM with attention‑control tools, (2) a concrete set of robot‑centric functions—including on‑demand vision queries, continuous gaze primitives, exploratory sweeps, and query‑driven retrieval—plus a lightweight viewpoint memory, and (3) an evaluation protocol and scenario suite that jointly measure turn‑level tool‑call accuracy and user‑perceived interaction quality under controlled ablations.

Limitations are acknowledged. The current implementation is tied to a specific robot platform with an egocentric RGB camera and 6‑DoF head; extending to other sensor modalities (depth, lidar) or mobile bases would require additional tool definitions. The reliance on the LLM’s internal policy for tool selection can produce over‑ or under‑calling, especially when the model’s confidence is misaligned with the task. Moreover, VLM‑based retrieval degrades when the query is ambiguous or the stored views lack sufficient coverage.

Future work suggested includes learning a reinforcement‑learning policy that fine‑tunes tool‑calling decisions based on interaction outcomes, integrating multimodal feedback (haptics, ambient sound) into the perception loop, and scaling the evaluation to larger, more diverse user populations. Overall, the study demonstrates that real‑time multimodal LLMs, when equipped with structured tool‑calling for active perception, provide a practical pathway toward fluid, situated human‑robot conversation in everyday environments.