Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration
Search-integrated reasoning enables language agents to transcend static parametric knowledge by actively querying external sources. However, training these agents via reinforcement learning is hindered by the multi-scale credit assignment problem: existing methods typically rely on sparse, trajectory-level rewards that fail to distinguish between high-quality reasoning and fortuitous guesses, leading to redundant or misleading search behaviors. To address this, we propose Search-R2, a novel Actor-Refiner collaboration framework that enhances reasoning through targeted intervention, with both components jointly optimized during training. Our approach decomposes the generation process into an Actor, which produces initial reasoning trajectories, and a Meta-Refiner, which selectively diagnoses and repairs flawed steps via a ‘cut-and-regenerate’ mechanism. To provide fine-grained supervision, we introduce a hybrid reward design that couples outcome correctness with a dense process reward quantifying the information density of retrieved evidence. Theoretically, we formalize the Actor-Refiner interaction as a smoothed mixture policy, proving that selective correction yields strict performance gains over strong baselines. Extensive experiments across various general and multi-hop QA datasets demonstrate that Search-R2 consistently outperforms strong RAG and RL-based baselines across model scales, achieving superior reasoning accuracy with minimal overhead.
💡 Research Summary
Search‑R2 tackles a fundamental obstacle in training search‑integrated reasoning agents with reinforcement learning: the multi‑scale credit assignment problem. Existing approaches rely on sparse, trajectory‑level rewards (e.g., final answer correctness) that cannot distinguish whether a successful answer resulted from high‑quality reasoning or from lucky guesses and redundant searches. Consequently, agents waste effort on unnecessary queries and learning becomes sample‑inefficient.
The paper introduces a two‑component framework composed of an Actor and a Meta‑Refiner. The Actor is a standard language model policy (πₗ) that generates reasoning chains interleaved with tool calls (
The Meta‑Refiner consists of:
-
Discriminator (π_d) – a binary gate that estimates the global coherence of the whole trajectory with respect to the question. If the probability exceeds a threshold τ, the trajectory is accepted; otherwise it is flagged for repair.
-
Trimmer (π_h) – a learned module that pinpoints the earliest step k where the reasoning deviates (the “root cause”). The prefix up to step k is kept, the suffix is discarded, and a new suffix is regenerated by the same Actor policy conditioned on the retained prefix. This “cut‑and‑regenerate” operation repairs only the faulty part, preserving useful intermediate work and dramatically improving sample efficiency compared with full‑trajectory rejection.
The interaction of Actor and Refiner defines a smoothed mixture policy q(y|x). The authors prove (Theorem 1) that, under mild conditions on the discriminator’s acceptance probability, the expected reward of q strictly dominates that of the pure Actor policy, establishing a theoretical guarantee for selective correction.
To provide dense supervision, a hybrid reward is defined:
- Outcome reward r_outcome(y) = I(a_pred = a_gold) (Exact Match).
- Process reward r_process(y) = (1/M) Σ_i u_i, where u_i ∈ {0,1} indicates whether retrieved chunk i is useful, as judged by an external evaluator.
The total reward is R(y) = r_outcome(y) · (1 + r_process(y)). This formulation ensures that high‑quality evidence is rewarded only when the final answer is correct, preventing reward hacking by simply retrieving many documents.
Training uses Group Relative Policy Optimization (GRPO). For each input, a batch of G trajectories is sampled from q(y|x). The hybrid rewards are group‑normalized to compute advantage estimates, and the loss combines clipped policy‑ratio terms with a KL‑regularization that keeps the refined policy close to the base Actor. This joint optimization updates the shared parameters θ of both Actor and Refiner end‑to‑end, solving the credit‑assignment problem across both scales.
Empirical evaluation spans three model sizes (7 B, 13 B, 32 B) and seven benchmarks, including open‑domain QA (NaturalQuestions, TriviaQA) and multi‑hop QA (HotpotQA, Musique). Search‑R2 consistently outperforms strong baselines such as RAG‑Fusion, ReAct, Self‑Ask, and recent RL‑based search agents, achieving 2–5 percentage‑point gains in exact‑match accuracy. Moreover, the average number of search queries per question drops by ~18 %, demonstrating more efficient use of the external knowledge source. Ablation studies confirm that both the discriminator‑driven acceptance and the trimmer‑driven cut‑and‑regenerate are essential; removing either component reduces performance to that of baseline rejection sampling.
Limitations discussed include the shared‑parameter design of Actor and Refiner (potentially limiting the Refiner’s expressive power), reliance on an external evaluator for process rewards (which incurs labeling cost), and focus on text‑only retrieval. Future work proposes dedicated Refiner networks, LLM‑based automatic evaluators, and extensions to multimodal retrieval.
In summary, Search‑R2 presents a principled, theoretically grounded, and empirically validated solution to the multi‑scale credit assignment dilemma in search‑integrated reasoning. By coupling fine‑grained error localization with selective regeneration and a hybrid reward signal, it enables language agents to reason more accurately while using external search resources more judiciously, marking a significant step toward robust, knowledge‑augmented AI systems.
Comments & Academic Discussion
Loading comments...
Leave a Comment