ToolRLA: Multiplicative Reward Decomposition for Tool-Integrated Agents

Tool-integrated agents that interleave reasoning with API calls are promising for complex tasks, yet aligning them for high-stakes, domain-specific deployment remains challenging: existing reinforcement learning approaches rely on coarse binary rewards that cannot distinguish tool selection errors from malformed parameters. We present ToolRLA, a three-stage post-training pipeline (SFT -> GRPO -> DPO) for domain-specific tool agents. The core contribution is a fine-grained reward function with multiplicative correctness decomposition spanning four dimensions – format validity, tool selection, parameter accuracy, and regulatory compliance – that encodes domain priority orderings as inductive biases in the reward landscape. Deployed on a financial advisory copilot (80+ advisors, 1,200+ daily queries), ToolRLA achieves over three months: a 47% improvement in task completion rate (62%->91%), a 63% reduction in tool invocation errors (38%->14%), and a 93% reduction in regulatory violations (12%->0.8%), within sub-2-second latency. Ablation studies show the multiplicative reward design accounts for 7 percentage points of improvement over additive alternatives. Generalization is further validated on ToolBench and API-Bank.

💡 Research Summary

ToolRLA addresses the pressing problem of aligning tool‑integrated large language model (LLM) agents for high‑stakes, domain‑specific deployments. Existing approaches either rely on multi‑model pipelines that cascade errors or on reinforcement learning (RL) with coarse binary rewards that cannot differentiate between a wrong tool selection, malformed parameters, or a regulatory violation. Both issues lead to low task success rates and unsafe behavior in regulated environments such as financial advisory.

The authors propose a three‑stage post‑training pipeline: (1) Supervised Fine‑Tuning (SFT) on 4.2 K sandbox‑verified trajectories to teach basic tool‑calling skills; (2) Group‑based Reinforcement Learning with a novel Multiplicative Reward Decomposition (GRPO) that evaluates each trajectory along four dimensions—format validity (R_fmt), correctness (R_cor), efficiency (R_eff), and compliance (R_cpl); and (3) Direct Preference Optimization (DPO) to capture “gray‑area” compliance signals that are hard to encode as explicit rules.

The core contribution is the fine‑grained reward function. R_fmt is a binary gate that ensures the model’s output is syntactically valid JSON with correct field names. R_cor is computed as the product of three sub‑scores: S_name (tool‑name correctness), S_comp (coverage of required tools), and S_acc (parameter accuracy measured in the sandbox). This multiplicative “veto” logic forces the entire correctness score to zero if the tool name is wrong, regardless of how well the parameters are formed. R_eff penalizes extra API calls by comparing the actual number of invocation steps to the minimal annotated optimal length. R_cpl imposes a large negative penalty (λ = 10) for any regulatory breach, guaranteeing that compliance dominates the other objectives (compliance ≫ correctness ≫ efficiency). The total reward is the sum of these four components, but the multiplicative structure inside R_cor is what differentiates ToolRLA from additive baselines.

GRPO replaces PPO’s value network with a group‑normalized advantage estimator. For each query, eight complete trajectories are sampled, executed in a sandbox, and scored with the above reward. The advantage is the trajectory’s reward minus the group mean, normalized by the group standard deviation. This approach halves GPU memory usage and provides stable gradients without learning a separate critic. Ablation experiments show that replacing the multiplicative composition with an additive one reduces task‑completion improvement by 7 percentage points, confirming the importance of the veto mechanism.

DPO is introduced because many compliance violations are subtle (e.g., implied forecasts) and cannot be captured by the hard‑coded regex + classifier pipeline used for R_cpl. The authors collect 2,038 preference pairs from 2,500 compliance‑sensitive production queries, where domain experts label one response as “chosen” and another as “rejected.” The DPO loss directly maximizes the likelihood of the chosen response over the rejected one, using the GRPO‑trained policy as a reference. This step reduces over‑refusal from 8 % to 1.5 % after adding a modest number of “helpful > over‑cautious” pairs and tuning the β hyperparameter to 0.2.

The system is deployed as a single‑model ReAct agent that interleaves Thought, Action, and Observation steps. Fifteen atomic tools and five composite tools are exposed via JSON‑Schema definitions. Composite tools bundle multiple atomic calls, reducing average invocation rounds from 4.2 to 2.8. Hallucination defenses (prompt‑level tool enumeration, runtime tool‑name validation, and a small set of error‑recovery demonstrations) cut hallucinated tool calls from ~8 % to <1 % after GRPO training.

Real‑world evaluation on a financial advisory copilot serving over 80 advisors and 1,200 daily queries yields dramatic improvements over a three‑month period: Task Completion Rate (TCR) rises from 62 % to 91 % (+47 pp), Tool Invocation Error Rate (TIER) drops from 38 % to 14 % (‑63 pp), average latency falls from 2.8 s to 1.6 s (‑43 %), and regulatory violation rate falls from 12 % to 0.8 % (‑93 %). Customer satisfaction climbs from 3.1 to 4.3 out of 5. Ablation studies confirm that each component—SFT warm‑start, multiplicative R_cor, large λ compliance penalty, and DPO fine‑tuning—contributes measurably to these gains.

Generalization is validated on public benchmarks ToolBench and API‑Bank. Compared to state‑of‑the‑art tool‑calling models such as Gorilla, ToolLLM, and GPT‑4 function calling, ToolRLA achieves higher call accuracy and plan‑plus‑call success, demonstrating that the reward decomposition and three‑stage pipeline are not limited to the financial domain.

In summary, ToolRLA provides a practical blueprint for safely deploying LLM‑based tool agents in regulated settings. By encoding domain‑specific priority orderings directly into a multiplicative reward structure, eliminating the need for a value network via GRPO, and polishing compliance behavior with DPO, the framework achieves both high task performance and stringent safety guarantees. The approach is readily extensible to other high‑risk sectors such as healthcare, legal advisory, or autonomous systems where tool use and regulatory compliance are critical.