Architecting Large Action Models for Human-in-the-Loop Intelligent Robots

The realization of intelligent robots, operating autonomously and interacting with other intelligent agents, human or artificial, requires the integration of environment perception, reasoning, and action. Classic Artificial Intelligence techniques for this purpose, focusing on symbolic approaches, have long-ago hit the scalability wall on compute and memory costs. Advances in Large Language Models in the past decade (neural approaches) have resulted in unprecedented displays of capability, at the cost of control, explainability, and interpretability. Large Action Models aim at extending Large Language Models to encompass the full perception, reasoning, and action cycle; however, they typically require substantially more comprehensive training and suffer from the same deficiencies in reliability. Here, we show it is possible to build competent Large Action Models by composing off-the-shelf foundation models, and that their control, interpretability, and explainability can be effected by incorporating symbolic wrappers and associated verification on their outputs, achieving verifiable neuro-symbolic solutions for intelligent robots. Our experiments on a multi-modal robot demonstrate that Large Action Model intelligence does not require massive end-to-end training, but can be achieved by integrating efficient perception models with a logic-driven core. We find that driving action execution through the generation of Planning Domain Definition Language (PDDL) code enables a human-in-the-loop verification stage that effectively mitigates action hallucinations. These results can support practitioners in the design and development of robotic Large Action Models across novel industries, and shed light on the ongoing challenges that must be addressed to ensure safety in the field.

💡 Research Summary

The pursuit of truly intelligent robots requires a seamless integration of perception, reasoning, and action. Historically, the field has been divided between two paradigms: symbolic AI, which offers logical rigor but lacks scalability in complex environments, and neural-based Large Language Models (LLMs), which demonstrate unprecedented reasoning capabilities but suffer from “black-box” opacity, high computational costs, and the dangerous phenomenon of “action hallucinations.” This paper proposes a groundbreaking neuro-symbolic architecture designed to bridge this gap, creating a reliable and verifiable Large Action Model (LAM) for human-in-the-loop robotics.

The core innovation of this research lies in the “Symbolic Wrapper” strategy. Rather than allowing an LLM to generate direct, unverified execution commands for a robot—which could lead to catastrophic physical errors—the authors restrict the LLM’s output to an intermediate symbolic representation using the Planning Domain Definition Language (PDDL). In this framework, the robot’s operational domain (the set of possible actions) is statically predefined and hardcoded, ensuring that the robot cannot attempt actions outside its physical capabilities. The LLM’s role is strategically limited to interpreting the current environmental state and defining the target goals in a structured PDDL format.

The architecture employs a deterministic solver, such as Fast Downward, to process the LLM-generated PDDL. This stage acts as a critical safety gate: if the LLM hallucinates an impossible goal or an inconsistent state, the solver will fail to find a valid plan, thereby preventing the erroneous instruction from ever reaching the physical actuators. This creates a software-level safety net that mitigates the risks inherent in neural-based reasoning.

Furthermore, the researchers formally integrated a “Human-in-the-loop” verification stage into the workflow. By providing an interface where users can review and manually edit the generated PDDL code or task sequences before execution, the system achieves high levels of transparency and user trust. This design ensures that even in complex or high-stakes scenarios, human oversight remains a central component of the autonomy loop.

Crucially, this intelligent behavior is achieved without the need for massive, end-to-end retraining of large-scale models. Instead, the system achieves high performance by composing off-the-shelf foundation models—including Vision-Language Models (VLMs) and LLMs—into a modular, ROS2-based framework. This approach is highly resource-efficient and provides a scalable blueprint for deploying intelligent, safe, and interpretable robots across various industrial sectors, from logistics to domestic service. The paper concludes that neuro-symbolic synthesis is a viable and powerful path toward achieving dependable autonomy in the next generation of intelligent robotic agents.