STAC: When Innocent Tools Form Dangerous Chains to Jailbreak LLM Agents

As LLMs advance into autonomous agents with tool-use capabilities, they introduce security challenges that extend beyond traditional content-based LLM safety concerns. This paper introduces Sequential Tool Attack Chaining (STAC), a novel multi-turn attack framework that exploits agent tool use. STAC chains together tool calls that each appear harmless in isolation but, when combined, collectively enable harmful operations that only become apparent at the final execution step. We apply our framework to automatically generate and systematically evaluate 483 STAC cases, featuring 1,352 sets of user-agent-environment interactions and spanning diverse domains, tasks, agent types, and 10 failure modes. Our evaluations show that state-of-the-art LLM agents, including GPT-4.1, are highly vulnerable to STAC, with attack success rates (ASR) exceeding 90% in most cases. The core design of STAC’s automated framework is a closed-loop pipeline that synthesizes executable multi-step tool chains, validates them through in-environment execution, and reverse-engineers stealthy multi-turn prompts that reliably induce agents to execute the verified malicious sequence. We further perform defense analysis against STAC and find that existing prompt-based defenses provide limited protection. To address this gap, we propose a new reasoning-driven defense prompt that achieves far stronger protection, cutting ASR by up to 28.8%. These results highlight a crucial gap: defending tool-enabled agents requires reasoning over entire action sequences and their cumulative effects, rather than evaluating isolated prompts or responses.

💡 Research Summary

The paper introduces a novel security threat for tool‑enabled large language model (LLM) agents called Sequential Tool Attack Chaining (STAC). Unlike traditional jailbreaks that aim to elicit unsafe text, STAC exploits the agent’s ability to call external tools. An attacker constructs a chain of tool calls where each intermediate step appears benign and passes safety checks, but the final step produces a harmful effect that only becomes apparent when the entire sequence is executed.

To study this threat, the authors build an automated pipeline consisting of five components: (1) a Generator that proposes a sequence of 2–6 tool calls given environment metadata and a failure mode; (2) a Verifier that executes each call in a sandboxed environment, observes the output, and iteratively revises the chain until every step is confirmed executable; (3) a Prompt Writer that reverse‑engineers a set of user prompts that logically lead the agent to invoke the verified intermediate calls; (4) a Planner that, during real‑world attack execution, adaptively crafts jailbreak prompts based on the agent’s responses and environment feedback to finally trigger the malicious terminal call; and (5) a Judge that scores prompt harmlessness, goal progress, and agent helpfulness after each turn.

Using this framework, the authors automatically generate and validate 483 distinct STAC cases across 1,352 user‑agent‑environment interaction instances. The benchmark spans diverse domains (banking, travel, workspace automation, web navigation), multiple simulated environments (SHADE‑Arena, Agent‑SafetyBench), and ten predefined failure modes (e.g., missing permissions, parameter errors).

Evaluation on a suite of twelve LLM agents—including proprietary frontier models such as GPT‑4.1, Claude‑3, and open‑source Llama‑2‑Chat—shows alarmingly high attack success rates (ASR) exceeding 90 % for most agents. Even models that incorporate strong safeguards against conventional jailbreaks are vulnerable because STAC distributes malicious intent across several turns, evading single‑turn safety checks.

The paper also assesses existing prompt‑based defenses (e.g., “refuse tool use”, “risk‑aware prompting”). These baselines achieve only modest reductions in ASR (5–12 %). To address the gap, the authors propose a new reasoning‑driven defense prompt that forces the agent to perform a harm‑benefit analysis before executing any tool call. In experiments this defense lowers ASR by up to 28.8 % (average ASR ≈ 58.6 %), demonstrating a significant but still incomplete mitigation.

Key contributions are: (1) formal definition of the STAC threat model; (2) an end‑to‑end automated framework for generating, verifying, and executing multi‑turn tool‑chain attacks; (3) a publicly released benchmark of 483 validated STAC scenarios; (4) a comprehensive empirical study revealing severe vulnerabilities in state‑of‑the‑art agents; and (5) a novel harm‑benefit reasoning defense that outperforms prior prompt‑only methods.

The findings highlight that securing tool‑enabled agents requires safety mechanisms that reason over entire action sequences and their cumulative impact, rather than evaluating isolated prompts or tool calls. This work opens a new research direction for building robust guardrails, auditing tool‑call pipelines, and designing meta‑reasoning capabilities that can detect and block malicious tool‑chain behavior before it materializes.