Multimodal Large Language Models for Real-Time Situated Reasoning

In this work, we explore how multimodal large language models can support real-time context- and value-aware decision-making. To do so, we combine the GPT-4o language model with a TurtleBot 4 platform simulating a smart vacuum cleaning robot in a home. The model evaluates the environment through vision input and determines whether it is appropriate to initiate cleaning. The system highlights the ability of these models to reason about domestic activities, social norms, and user preferences and take nuanced decisions aligned with the values of the people involved, such as cleanliness, comfort, and safety. We demonstrate the system in a realistic home environment, showing its ability to infer context and values from limited visual input. Our results highlight the promise of multimodal large language models in enhancing robotic autonomy and situational awareness, while also underscoring challenges related to consistency, bias, and real-time performance.

💡 Research Summary

The paper investigates the feasibility of deploying multimodal large language models (MLLMs) for real‑time, value‑aware decision‑making in domestic robotics. By integrating OpenAI’s GPT‑4o, which can process both images and text, with a TurtleBot 4 platform, the authors create a “value‑aware vacuum cleaner” that interprets visual scenes, infers human activities, detects the presence of people or pets, and reasons about user preferences, social norms, and safety before deciding whether to start cleaning, wait, or return to its docking station.

Motivation and Background
Traditional home cleaning robots rely on pre‑programmed schedules, simple obstacle avoidance, and limited learning. They lack the ability to understand the contextual nuances of a household—such as whether a resident is watching a movie and needs silence, or whether a pet is moving around and could be frightened by the robot’s noise. Recent advances in large language models have shown strong commonsense reasoning and the capacity to generate human‑readable explanations, suggesting they could fill this gap if combined with multimodal inputs.

System Architecture
The hardware consists of a TurtleBot 4 equipped with an OAK‑D‑PRO RGB‑D camera. During the “observation” state the robot rotates 180°, capturing ten images at a rate of one frame per second. These images are sent to GPT‑4o, which first extracts salient elements (people, pets, activities) and then performs a step‑by‑step reasoning process guided by a carefully crafted system prompt. The prompt defines the robot’s role, its overarching objective (“maintain cleanliness while respecting the homeowner’s values”), and detailed descriptions of the three operational modes (observation, cleaning, docking). The model is instructed to consider value alignment, temporal context (current time is supplied), potential consequences of each action, and to produce a rationale trace that is fed back into the model for a final decision. The final output includes a concise, user‑friendly explanation of the chosen action.

The robot’s control logic is implemented in ROS 2, with a Python node handling communication with the camera and the LLM, and a PyQt5 GUI displaying the reasoning trace and allowing manual overrides. In cleaning mode the robot moves forward until proximity sensors detect an obstacle, then turns randomly; during cleaning it continues to capture images every 0.5 seconds to monitor for value violations that might require an immediate transition back to observation. Docking mode currently stops the robot at its charging station, but the authors note that future extensions could allow the robot to relocate to a quieter room before charging.

Experimental Evaluation
Two evaluation strategies were employed. First, a controlled image‑only test using selected frames from the YouHome Activities of Daily Living dataset demonstrated that GPT‑4o can reliably detect people and infer activities such as “watching TV” or “using a phone,” even when the scene is empty or rapidly changing. Second, a live deployment in a real living‑room setting examined three scenarios: (1) a person watching a movie, (2) a person using a smartphone, and (3) a dog moving around. In the movie‑watching case the robot correctly inferred that noise would be disruptive and deferred cleaning, citing the noise as justification. When the person was on a phone, the model judged that silence was not essential and initiated cleaning. With the dog present, the robot recognized potential safety and stress concerns for the pet and chose to wait, explicitly mentioning the pet’s wellbeing in its rationale.

Limitations and Discussion
The authors acknowledge several challenges. Real‑time performance is constrained by image transmission, processing, and the latency of GPT‑4o calls, leading to delays on the order of one second—acceptable for low‑frequency decisions but problematic for fast‑reacting tasks. Longer prompts or chain‑of‑thought techniques, while potentially improving reasoning depth, were avoided due to additional latency. Consistency is another issue: the stochastic nature of LLM outputs can produce different decisions for identical inputs, which is undesirable in safety‑critical robotics. Biases inherited from the pre‑training data may cause the model to over‑estimate the need for silence or to be overly cautious around pets. Privacy concerns arise from sending in‑home images to a cloud service; the authors suggest future work on locally hosted models and user‑provided preference profiles to mitigate this.

Conclusion and Future Work
The study demonstrates that multimodal LLMs can endow domestic robots with a rudimentary form of situational awareness and value‑aligned decision making, moving beyond pure task execution toward socially aware behavior. However, to transition from prototype to reliable household assistant, further research is needed on latency reduction (e.g., edge inference), prompt engineering for consistency, bias mitigation through user‑specific conditioning, and privacy‑preserving architectures. The authors envision extensions that integrate additional modalities (audio, temperature), richer user preference interfaces, and more sophisticated state machines that can dynamically re‑plan routes based on real‑time value assessments.