When Are Two RLHF Objectives the Same?

The preference optimization literature contains many proposed objectives, often presented as distinct improvements. We introduce Opal, a canonicalization algorithm that determines whether two preference objectives are algebraically equivalent by producing either a canonical form or a concrete witness of non-equivalence. Applying Opal reveals that many widely used methods optimize the same underlying objective, while others are provably distinct. For example, batch normalization can cause the same response pair to receive different gradients depending on batch composition. We identify a small set of structural mechanisms that give rise to genuinely different objectives; most remaining differences are reparameterizations.

💡 Research Summary

This paper tackles a fundamental yet under‑explored question in the rapidly growing field of Reinforcement Learning from Human Feedback (RLHF): when do two seemingly different preference‑optimization objectives actually represent the same underlying mathematical problem? To answer this, the authors introduce Opal, a canonicalization algorithm that either reduces any pair‑margin‑based objective to a unique normal form or produces a concrete witness proving non‑equivalence.

The authors begin by reviewing the landscape of RLHF objectives. Starting from the classic pipeline—train a reward model from human comparisons and then fine‑tune a language model with PPO—they discuss Direct Preference Optimization (DPO) and its many variants (IPO, SimPO, ORPO, KTO, etc.). Game‑theoretic formulations such as SPPO and Nash‑MD are also covered, as are newer approaches like GRPO (which normalizes advantages within a batch) and token‑level or trajectory‑level methods (RTO, PPO‑RLHF). All of these papers claim improvements, but it is unclear whether the claimed gains stem from genuinely new objectives or from mere re‑parameterizations.

Opal’s theoretical foundation rests on the observation that any pair‑wise margin‑based loss can be expressed as a composition of three primitive operations:

1. **Add