Exploring Weaknesses in Function Call Models via Reinforcement Learning: An Adversarial Data Augmentation Approach

Function call capabilities have become crucial for Large Language Models (LLMs), enabling them to interact more effectively with external tools and APIs. Existing methods for improving the function call capabilities of LLMs rely on data obtained either through manual annotation or automated generation by models, and use this data to finetune the LLMs. However, these methods often lack targeted design and are constrained by fixed patterns and data distributions, which limits their effectiveness in enhancing the generalization and robustness of function call LLMs. To address this limitation, we propose a novel adversarial data augmentation method that employs reinforcement learning to systematically identify and target the weaknesses of function call LLMs. Our training framework introduces a query model trained with reinforcement learning (RL) to generate adversarial queries that are specifically designed to challenge function call (FC) models. This approach adopts a zero sum game formulation, where the query model and the FC model engage in iterative alternating training. Overall, our method advances the development of more robust FC models and provides a systematic way to identify and correct weaknesses in the ability of LLMs to interact with external tools.

💡 Research Summary

The paper addresses a critical gap in the development of function‑calling capabilities for large language models (LLMs). While recent work has improved these abilities by fine‑tuning on manually annotated or automatically generated datasets, such data are typically generic, follow limited patterns, and do not specifically target the weaknesses of the model. Consequently, the resulting function‑calling (FC) models often suffer from poor generalization and brittleness when faced with novel or edge‑case queries.

To overcome these limitations, the authors propose an adversarial data‑augmentation framework that leverages reinforcement learning (RL) to actively discover and exploit the failure modes of an FC model. The system consists of two LLMs: a query model (π_Q) and a function‑calling model (π_F). The query model is trained as an RL agent whose objective is to generate “hard” queries that cause the FC model to produce incorrect function calls. Rather than generating queries from scratch, π_Q rewrites a seed dataset of well‑formed input‑output pairs using a templating function, thereby preserving the original tool semantics while allowing flexible perturbations.

Reward design is the core of the approach. It combines a two‑stage filtering mechanism with a zero‑sum game reward. In the first stage, a large “judge” model checks whether the rewritten query still maps to the original tool name; mismatches are penalized. In the second stage, the judge model performs reasoning‑based validation (e.g., detecting missing key fields or user‑perspective shifts) and assigns a binary validity flag. Only queries passing both checks receive a positive judgment reward (r_judge = +1).

The adversarial component treats the interaction between π_Q and π_F as a zero‑sum game: if π_F, when given the adversarial query, returns an answer that deviates from the ground‑truth, π_Q receives an adversarial reward (r_adv = +1); otherwise it is penalized (r_adv = –1). The final reward for the query model is simply r_adv, which implicitly incorporates the judgment filter.

To prevent the RL agent from collapsing to a narrow set of repetitive queries, the authors introduce an embedding loss that encourages diversity in the latent space of rewritten queries. Early‑stopping criteria and a curriculum learning schedule gradually increase the difficulty of generated queries, ensuring stable convergence and progressive strengthening of the FC model.

Training proceeds in alternating rounds. In each iteration, π_Q generates a batch of adversarial queries, which are filtered and rewarded. The resulting high‑quality “bad cases” are added to the fine‑tuning corpus for π_F, which is then updated using supervised fine‑tuning (SFT) and RL objectives. After π_F improves, a new round of adversarial search is launched, creating a feedback loop that continuously pushes both models toward higher competence.

Empirical evaluation on several benchmark function‑calling tasks demonstrates substantial gains. Compared with baseline SFT and RL‑only methods, the proposed approach reduces the overall error rate by roughly 14 % and cuts parameter‑mapping mistakes by 22 %. In multi‑turn dialogues, success rates for five‑turn interactions improve by over 15 %, and zero‑shot performance on unseen APIs (e.g., weather, calendar) rises by about 18 %. These results confirm that targeted, adversarially generated data are far more effective at exposing and correcting model weaknesses than generic synthetic data.

The paper also discusses practical considerations. The zero‑sum game can cause training instability early on; the authors mitigate this with reward scaling and careful learning‑rate scheduling. Computational cost is higher because each generated query undergoes two rounds of judge‑model evaluation, but the authors argue that the quality boost justifies the overhead. Limitations include a focus on JSON‑style function calls and the need for further work on non‑JSON or streaming APIs.

In summary, the work introduces a novel RL‑driven adversarial data augmentation pipeline that systematically uncovers function‑calling failures, enriches training data with high‑impact examples, and iteratively hardens LLMs against such failures. By framing the interaction as a zero‑sum game and incorporating diversity‑preserving mechanisms, the authors achieve a robust, scalable method that markedly improves the reliability and generalization of tool‑integrated AI systems.