Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning

Reinforcement learning from human feedback (RLHF) has emerged as a key technique for aligning the output of large language models (LLMs) with human preferences. To learn the reward function, most existing RLHF algorithms use the Bradley-Terry model, which relies on assumptions about human preferences that may not reflect the complexity and variability of real-world judgments. In this paper, we propose a robust algorithm to enhance the performance of existing approaches under such reward model misspecifications. Theoretically, our algorithm reduces the variance of reward and policy estimators, leading to improved regret bounds. Empirical evaluations on LLM benchmark datasets demonstrate that the proposed algorithm consistently outperforms existing methods, with 77-81% of responses being favored over baselines on the Anthropic Helpful and Harmless dataset. The code is available at https://github.com/VRPO/VRPO.

💡 Research Summary

This paper addresses a fundamental weakness in current reinforcement learning from human feedback (RLHF) pipelines for large language models (LLMs). Most RLHF methods rely on the Bradley‑Terry (BT) model to convert pairwise human preferences into a scalar reward function. The BT model assumes transitivity, context‑independence, and perfect rationality of human judgments—assumptions that are frequently violated in practice. When these assumptions fail, the learned reward model becomes misspecified, leading to biased or high‑variance estimates and sub‑optimal policies.

To mitigate this problem, the authors propose Variance‑Reduced Preference Optimization (VRPO), a flexible augmentation that can be attached to a wide range of existing RLHF algorithms. VRPO introduces an auxiliary “reward‑free” preference model that is trained on a massive pool of unlabeled response pairs generated by a known reference policy (π_ref). Because π_ref is either the pre‑trained LLM or a supervised‑fine‑tuned (SFT) version, it can be considered well‑specified for the data distribution of interest. By sampling unlimited (prompt, response₁, response₂) triples from π_ref without human labels, VRPO creates a semi‑supervised learning setting: the auxiliary model learns the structure of human preferences from the unlabeled data, while the primary reward model continues to be trained on the limited labeled pairs.

The key technical contribution is a variance‑reduction estimator that combines the predictions of the primary reward model r_θ and the auxiliary preference model q_φ. For labeled pairs, the standard cross‑entropy loss is used to fit r_θ. For unlabeled pairs, the algorithm leverages q_φ’s output together with the known sampling distribution of π_ref to construct an unbiased correction term. This blended estimator reduces both the variance and the mean‑squared error (MSE) of the reward estimate compared with conventional RLHF.

Theoretical analysis (Theorem 6.2) shows that, under model misspecification, the MSE of the VRPO estimator is strictly smaller than that of the baseline estimator, while Theorem 6.3 establishes a tighter bound on the sub‑optimality gap (regret) of the resulting policy. Importantly, when π_ref is “well‑specified” (i.e., close to the true optimal policy), the auxiliary model provides near‑perfect information, and the variance reduction becomes asymptotically negligible, yielding an essentially unbiased estimator.

Empirically, the authors evaluate VRPO on several LLM benchmark datasets, most notably the Anthropic Helpful‑Harmless (HH) dataset. They integrate VRPO with Proximal Policy Optimization (PPO) and compare against nine strong baselines, including both two‑stage (explicit reward model) and one‑stage (implicit reward) RLHF methods. Across multiple runs, VRPO‑enhanced policies are preferred over baselines in 53‑98% of pairwise human evaluations, with an average preference rate of 77‑81%. Additional ablation studies deliberately corrupt the reward model (e.g., replacing the sigmoid activation with a tanh) and the reward function itself (using a non‑parametric form that cannot be captured by the chosen parametric family). In all misspecification scenarios, VRPO maintains a clear advantage, demonstrating robustness to both reward‑model and preference‑model errors.

The paper situates its contribution within three strands of related work: (1) reward‑based RLHF that typically uses BT or its extensions; (2) preference‑based RLHF that avoids explicit rewards but still assumes a particular preference structure; and (3) robust RLHF methods that focus on noisy or biased feedback, heterogeneous teachers, or distribution shift. VRPO differs by directly addressing internal model misspecification while preserving computational efficiency; the auxiliary model adds negligible overhead because it does not require reward evaluation during policy optimization.

Limitations are acknowledged. VRPO’s effectiveness hinges on the availability of a well‑specified reference policy; if the deployment distribution diverges substantially from the data used to train π_ref, the auxiliary model’s guidance may become less reliable. Moreover, generating the large unlabeled sample set incurs non‑trivial compute costs, especially for very large LLMs. The authors suggest future work on adaptive reference‑policy updates, multi‑domain feedback integration, and more efficient sampling strategies.

In conclusion, VRPO offers a principled, theoretically grounded, and empirically validated solution to the problem of reward‑model misspecification in RLHF. By leveraging semi‑supervised variance reduction, it improves sample efficiency, reduces estimator variance, and yields policies that align more closely with human preferences, making it a valuable addition to the toolkit for aligning large language models.