ARTIS: Agentic Risk-Aware Test-Time Scaling via Iterative Simulation

Current test-time scaling (TTS) techniques enhance large language model (LLM) performance by allocating additional computation at inference time, yet they remain insufficient for agentic settings, where actions directly interact with external environments and their effects can be irreversible and costly. We propose ARTIS, Agentic Risk-Aware Test-Time Scaling via Iterative Simulation, a framework that decouples exploration from commitment by enabling test-time exploration through simulated interactions prior to real-world execution. This design allows extending inference-time computation to improve action-level reliability and robustness without incurring environmental risk. We further show that naive LLM-based simulators struggle to capture rare but high-impact failure modes, substantially limiting their effectiveness for agentic decision making. To address this limitation, we introduce a risk-aware tool simulator that emphasizes fidelity on failure-inducing actions via targeted data generation and rebalanced training. Experiments on multi-turn and multi-step agentic benchmarks demonstrate that iterative simulation substantially improves agent reliability, and that risk-aware simulation is essential for consistently realizing these gains across models and tasks.

💡 Research Summary

The paper “ARTIS: Agentic Risk‑Aware Test‑Time Scaling via Iterative Simulation” addresses a critical gap in current test‑time scaling (TTS) methods for large language models (LLMs). Existing TTS techniques improve answer quality by allocating extra computation during inference, but they assume that intermediate reasoning steps are cheap, reversible, and have no real‑world consequences. In agentic settings—where LLMs act as autonomous agents that invoke tools, APIs, or physical devices—each action can permanently alter the environment, incur costs, or cause safety‑critical failures. Consequently, a new form of TTS is needed that scales computation over actions rather than over static reasoning traces.

Core Idea
ARTIS (Agentic Risk‑Aware Test‑time Scaling via Iterative Simulation) introduces a three‑stage inference pipeline that separates exploration from commitment:

1. Iterative Simulation Loop – Given the current conversation context, available tools, and user request, the agent generates up to N candidate action plans. Two generation strategies are explored:

   • Sequential iteration: each simulated attempt can observe the outcomes of previous attempts, enabling adaptive refinement but incurring higher latency and longer context windows.
   • Parallel iteration: all attempts are sampled independently, allowing massive parallelism at the cost of potential redundancy.

2. Self‑Evaluation – After each simulated execution, a separate evaluation step (implemented by the same LLM) produces a binary correctness signal and natural‑language feedback describing why an attempt succeeded or failed. This feedback is fed back into subsequent attempts (in the sequential mode) or used for later summarization.

3. Summarization of Simulated Attempts – Rather than feeding all raw simulated trajectories into the final prompt, ARTIS compresses the diverse experiences into a concise high‑level recommendation (S). This summary acts as a risk‑averse guide for the final real‑world execution, reducing noise and preventing over‑fitting to any single noisy simulation.

The final step is a single committed execution in the real environment, conditioned on the original context and the summarised recommendation. By limiting real‑world actions to one pass, ARTIS eliminates the possibility of costly roll‑backs while still benefiting from extensive offline computation.

Risk‑Aware Simulator
A pivotal contribution is the identification that generic LLM‑based simulators are ill‑suited for agentic tasks. They tend to excel on average‑case predictions but systematically miss rare, high‑impact failure modes—precisely the scenarios where safety matters most. To remedy this, the authors construct a risk‑aware tool simulator:

• They generate targeted training data that over‑represents failure‑inducing tool calls (e.g., malformed code, out‑of‑bounds API parameters, security‑sensitive operations).
• The dataset is re‑balanced to give higher weight to these hard cases during fine‑tuning.
• The resulting simulator shows markedly higher fidelity on failure‑rich queries while maintaining comparable performance on benign cases.

Empirical results demonstrate that when a perfect simulator (i.e., one that reproduces the true environment) is available, increasing N yields substantial gains in task success rates, confirming the value of iterative simulation (Research Question R1). Conversely, using a naïve LLM‑based simulator leads to performance degradation, even below the no‑simulation baseline, confirming that current generic simulators cannot support R2.

When the risk‑aware simulator is employed, ARTIS consistently outperforms standard inference, conventional TTS baselines, and naïve simulation across two challenging benchmarks:

• BFCL‑v3 – a multi‑turn Python‑based environment where agents must orchestrate several tool calls.
• ACEBench – a suite of complex, multi‑step agentic tasks with diverse toolsets.

Across both benchmarks, ARTIS with the risk‑aware simulator improves accuracy by roughly 10–15 percentage points over the strongest baselines, and the gains are robust across model sizes (e.g., Qwen‑3‑8B, larger proprietary LLMs). Ablation studies show that (a) the summarization step prevents noisy simulations from harming final performance, and (b) sequential iteration yields higher reliability than parallel iteration when computational budget permits, due to its ability to learn from previous failures.

Implications and Future Directions
The work reframes test‑time scaling for LLMs as an action‑centric problem, aligning it with concepts from model‑predictive control, planning with world models, and safety‑critical AI. It highlights that the quality of the simulated environment is the bottleneck; without a simulator that faithfully predicts rare catastrophic outcomes, additional computation can be counterproductive.

Potential extensions include:

• Domain adaptation – bridging the gap between simulated and real environments via online fine‑tuning or meta‑learning.
• Multi‑modal simulation – incorporating visual or physical dynamics for embodied agents.
• Adaptive budgeting – dynamically deciding the number of simulated attempts based on confidence estimates.
• Hierarchical risk modeling – integrating explicit utility or cost functions to prioritize simulations that explore high‑risk regions of the action space.

In summary, ARTIS introduces a principled, risk‑aware test‑time scaling framework that leverages iterative simulation to improve the reliability of LLM‑driven agents in irreversible, high‑stakes environments. By coupling this with a simulator trained to emphasize failure modes, the authors demonstrate that substantial performance gains are achievable without sacrificing safety, paving the way for more trustworthy autonomous language‑model agents.