Chance-Constrained Inference for Hallucination Risk Control in Large Language Models

Large language models generate outputs stochastically and may produce fluent but invalid responses, including factual hallucinations. Existing mitigation strategies reduce average error rates but do not provide explicit control over the \emph{frequency} of such failures under repeated use. We formulate inference as a deployment-time risk control problem and introduce \emph{chance-constrained inference}, which directly bounds the probability of hallucinations among accepted generations. Hallucinations are modeled as stochastic constraint violations, and we show that confidence-based selective prediction does not, in general, imply probabilistic risk guarantees. To enforce chance constraints efficiently, we propose a sequential, anytime-valid inference procedure that adaptively certifies feasibility or infeasibility using finite samples, avoiding conservative fixed-sample bounds. Experiments on questions inspired by NaturalQuestions and controlled multi-hop question answering demonstrate reliable risk control, early detection of intrinsically infeasible inputs, and safe composition under repeated use, while confidence-based baselines fail to provide consistent guarantees.

💡 Research Summary

The paper “Chance-Constrained Inference for Hallucination Risk Control in Large Language Models” introduces a novel inference-time framework designed to provide explicit, statistical guarantees on the frequency of hallucinations (factually incorrect or unsupported outputs) when LLMs are used repeatedly. The core problem with existing mitigation techniques—such as retrieval-augmented generation, confidence-based filtering, or post-hoc verification—is that they reduce the average error rate but do not formally control how often failures occur under stochastic decoding and repeated use. This is critical for high-stakes or agentic applications where even low per-query risks can accumulate.

The authors recast LLM inference as a deployment-time risk management problem. They propose Chance-Constrained Inference (CCI), which directly bounds the conditional probability that an output presented to the user violates a predefined constraint (e.g., factual correctness). For a given input x and a user-specified risk budget ε(x), CCI ensures that P(H(x,y)=1 | A(x,y)=1) ≤ ε(x), where H(x,y) is a violation indicator and A(x,y) is the acceptance policy determining if output y is shown to the user.

A key insight is that hallucination is treated not as a deterministic error but as a stochastic event. Therefore, the goal shifts from eliminating every error to controlling their long-run frequency. To enforce this chance constraint efficiently, the paper devises a sequential, anytime-valid inference procedure. Instead of generating a fixed, large number of samples upfront, the system adaptively generates outputs, checking after each one whether the accumulated evidence certifies one of three states: Feasible (the risk budget can be satisfied with high confidence), Infeasible (the budget is provably violated), or Undecided. This allows for early stopping, especially for “infeasible” inputs where the model intrinsically has a high probability of hallucination, enabling safe abstention without wasteful computation.

The methodology is model-agnostic, requiring no retraining or fine-tuning of the base LLM. It operates purely at inference time using a verifier function H(x,y) that can encapsulate various checks (factuality, logical consistency, policy compliance). The sequential procedure employs anytime-valid concentration inequalities, providing valid statistical guarantees regardless of when the sampling stops, a significant advantage over conservative fixed-sample bounds.

Experiments are conducted on question-answering tasks inspired by NaturalQuestions and controlled multi-hop QA. The results demonstrate that CCI reliably controls risk at the specified levels (e.g., 1% or 5%), whereas confidence-based selective prediction baselines often violate or overly conservatively meet the budget. Furthermore, CCI successfully identifies and abstains on infeasible queries using far fewer samples than baseline methods. Finally, the paper shows that systems composed of CCI-accepted outputs maintain the risk guarantee under repeated use, a property termed “safe composition.”

In summary, this work makes a significant contribution by bringing the rigorous framework of chance-constrained optimization from stochastic control to LLM inference. It provides a principled, statistically sound method to enforce reliability guarantees on a per-input basis, enhancing the safety and trustworthiness of LLMs in deployment scenarios where controlling failure rates is paramount.