Improving Value-based Process Verifier via Structural Prior Injection

In the Large Language Model(LLM) reasoning scenario, people often estimate state value via Monte Carlo sampling. Though Monte Carlo estimation is an elegant method with less inductive bias, noise and errors are inevitably introduced due to the limited sampling. To handle the problem, we inject the structural prior into the value representation and transfer the scalar value into the expectation of a pre-defined categorical distribution, representing the noise and errors from a distribution perspective. Specifically, by treating the result of Monte Carlo sampling as a single sample from the prior ground-truth Binomial distribution, we quantify the sampling error as the mismatch between posterior estimated distribution and ground-truth distribution, which is thus optimized via distribution selection optimization. We test the performance of value-based process verifiers on Best-of-N task and Beam search task. Compared with the scalar value representation, we show that reasonable structural prior injection induced by different objective functions or optimization methods can improve the performance of value-based process verifiers for about 1$\sim$2 points at little-to-no cost. We also show that under different structural prior, the verifiers’ performances vary greatly despite having the same optimal solution, indicating the importance of reasonable structural prior injection.

💡 Research Summary

The paper addresses the problem of noisy and coarse value estimates in large‑language‑model (LLM) reasoning when state values are annotated via Monte Carlo sampling. Because each rollout must reach a final binary outcome, the number of samples (k) is often small, leading to a discrete set of possible value estimates (0, 1/k, 2/k, …, 1) and high variance especially when the true success probability p lies near 0.5. Existing work mitigates these issues by either applying a mean‑square‑error (MSE) loss to impose a “distance prior” that smooths the discrete values into a continuous space, or by using temporal‑difference updates that require many additional rollouts. Both approaches are costly or insufficient for LLM reasoning where each rollout is expensive.

The authors propose a “structural prior injection” framework that re‑interprets the scalar value as the expectation of a predefined categorical distribution. Specifically, they treat the k Monte Carlo rollouts as a single draw from a Binomial(k, p) distribution, where p is the ground‑truth success probability of the current state. The observed Monte Carlo estimate (\hat V) is thus a single sample from this Binomial distribution. The learning objective becomes the recovery of the underlying distribution rather than a point estimate, turning sampling error into a distribution‑mismatch problem.

To quantify how well a candidate posterior distribution matches the true Binomial prior, the paper introduces a novel metric called Statistics‑based Distance. This distance measures the divergence between two categorical distributions conditioned on the probability of sampling each category from the ground‑truth distribution. It serves as a guide for selecting a reasonable structural prior and for steering the optimization process toward posterior distributions that are close to the true Binomial.

Two concrete optimization strategies are explored. The first uses the traditional MSE loss but applied to the expectation of a categorical distribution whose support points are the same as the Binomial outcomes (i.e., ({0, 1/k, …, 1})). This “expectation regression” retains the distance prior while adding the structural prior that the distribution’s shape should resemble a Binomial. The second strategy replaces the MSE with a cross‑entropy (histogram) loss, directly training the verifier to output a categorical probability vector that approximates the full Binomial distribution. In both cases the verifier network (f_\theta) maps a question‑state pair ((q, s_t)) to a probability vector over the (k+1) bins.

Empirical evaluation is performed on two representative LLM reasoning tasks: (1) Best‑of‑N, where multiple candidate completions are generated and the verifier must rank them by value; and (2) Beam‑search, where the verifier predicts the value of each beam during search to guide pruning. Across both tasks, injecting a reasonable structural prior yields consistent improvements of roughly 1–2 percentage points in success metrics compared with a baseline scalar‑regression verifier. Ablation studies reveal that the choice of prior (e.g., number of bins, placement of support points) can cause large performance swings even when the optimal solution (the true p) is unchanged, underscoring the importance of careful prior design.

The paper also discusses limitations and future directions. The current formulation assumes a Binomial prior, which may be too simplistic for real LLM error patterns that exhibit over‑dispersion or correlation across rollouts. Extending the framework to richer priors such as Beta or Dirichlet‑multinomial distributions, or incorporating hierarchical Bayesian updates, could further reduce variance. Moreover, automating prior selection via meta‑learning or learning the prior jointly with the verifier is identified as a promising avenue.

In summary, the work reframes value estimation for LLM reasoning from a point‑wise regression problem into a distribution‑matching problem, introduces a principled distance metric for prior evaluation, and demonstrates that modest structural priors can yield measurable gains with negligible computational overhead. This opens a new line of research on probabilistic modeling of verification signals in LLM‑driven decision making.