PACE: Defying the Scaling Hypothesis of Exploration in Iterative Alignment for Mathematical Reasoning

Iterative Direct Preference Optimization has emerged as the state-of-the-art paradigm for aligning Large Language Models on reasoning tasks. Standard implementations (DPO-R1) rely on Best-of-N sampling (e.g., $N \ge 8$) to mine golden trajectories from the distribution tail. In this paper, we challenge this scaling hypothesis and reveal a counter-intuitive phenomenon: in mathematical reasoning, aggressive exploration yields diminishing returns and even catastrophic policy collapse. We theoretically demonstrate that scaling $N$ amplifies verifier noise and induces detrimental distribution shifts. To resolve this, we introduce \textbf{PACE} (Proximal Alignment via Corrective Exploration), which replaces brute-force mining with a generation-based corrective strategy. Operating with a minimal budget ($2<N<3$), PACE synthesizes high-fidelity preference pairs from failed explorations. Empirical evaluations show that PACE outperforms DPO-R1 $(N=16)$ while using only about $1/5$ of the compute, demonstrating superior robustness against reward hacking and label noise.

💡 Research Summary

The paper challenges the prevailing assumption in iterative Direct Preference Optimization (DPO) that larger sampling budgets (Best‑of‑N with N ≥ 8) inevitably yield better alignment for mathematical reasoning. The authors first provide a theoretical analysis showing that increasing N amplifies two detrimental effects.

1. False‑Positive Amplification – The verifier (reward model) is noisy, characterized by a defect rate ε. When the model’s intrinsic success probability is α, the probability that a selected “winner” is actually a false positive grows as more candidates are drawn. The marginal pass rate of newly added samples drops roughly as 1/N, so a larger N systematically introduces low‑quality, verifier‑hacked examples that mislead the policy.

2. Distributional Shift – Even with a perfect verifier, selecting the extreme tail of the generation distribution forces the policy to move far from its current distribution. Using a KL‑divergence lower bound, the authors show that the required shift grows with the gap between the current success rate α and a target rate η, leading to instability and potential policy collapse when N is large.

To address these issues, the authors propose PACE (Proximal Alignment via Corrective Exploration), a three‑phase framework that deliberately limits exploration to N ≈ 2 and leverages failed attempts as learning signals.

Phase I – Proximal Exploration: Two candidate solutions are sampled per prompt, probing the local variance of the current policy without venturing into the distribution tail.

Phase II – Hindsight Refinement with Quality Gating: When both candidates are incorrect, the model is prompted with its own error trace and the ground‑truth answer to generate a corrected reasoning path (y_fix). A strict consistency filter removes “rationalizations” that merely produce the right answer without logical justification.

Phase III – Contrastive Pair Construction: The corrected path (positive) and the original error (hard negative) form a proximal preference pair. Because the two trajectories share semantic structure but differ in logical validity, the resulting DPO gradient carries high information density, unlike the easy, semantically distant pairs produced by standard Best‑of‑N.

Empirical evaluation on GSM8K, MATH, and a noisy‑label variant (20 % corrupted ground truth) demonstrates that PACE matches or exceeds the performance of DPO‑R1 with N = 16 while using roughly one‑fifth of the compute. In noisy settings, DPO‑R1’s accuracy collapses, whereas PACE remains stable, confirming its robustness to verifier noise and label corruption.

The paper acknowledges limitations: the verifier remains a neural scorer and may still misjudge complex proofs; the hindsight correction relies on the model’s own capabilities, which could be weak in early training stages. Future work is suggested to integrate symbolic proof checkers or external theorem provers for higher‑fidelity correction and to explore multi‑step hindsight pipelines to mitigate error accumulation.

In summary, the work overturns the “more exploration = better alignment” dogma for mathematical reasoning, offering a principled, compute‑efficient alternative that prioritizes proximal, corrective learning over brute‑force mining of extreme samples.