Ranking-aware Reinforcement Learning for Ordinal Ranking

Ordinal regression and ranking are challenging due to inherent ordinal dependencies that conventional methods struggle to model. We propose Ranking-Aware Reinforcement Learning (RARL), a novel RL framework that explicitly learns these relationships. At its core, RARL features a unified objective that synergistically integrates regression and Learning-to-Rank (L2R), enabling mutual improvement between the two tasks. This is driven by a ranking-aware verifiable reward that jointly assesses regression precision and ranking accuracy, facilitating direct model updates via policy optimization. To further enhance training, we introduce Response Mutation Operations (RMO), which inject controlled noise to improve exploration and prevent stagnation at saddle points. The effectiveness of RARL is validated through extensive experiments on three distinct benchmarks.

💡 Research Summary

The paper introduces Ranking‑Aware Reinforcement Learning (RARL), a novel framework that unifies ordinal regression and learning‑to‑rank (L2R) within a single reinforcement‑learning (RL) objective. Traditional approaches treat regression and ranking as separate tasks, which leads to a trade‑off between predicting accurate continuous values and preserving the inherent order among categories. RARL resolves this by defining a verifiable reward function that simultaneously evaluates regression precision and ranking consistency, allowing a policy network to be directly optimized without a learned reward model. The core reward, Rfinal, is a weighted sum of three components: (1) a regression reward that grants non‑zero value only when the predicted ordinal value lies within a tolerance δ of the ground truth, (2) a ranking reward composed of length compliance, Kendall‑τ‑based consistency between regression‑derived and model‑generated permutations, and direct Kendall‑τ agreement with human annotations, and (3) a format reward ensuring the output follows the required JSON schema. By linearly combining these terms with tunable λ‑weights, RARL creates a bidirectional regularization where regression errors inform ranking updates and vice‑versa.

Policy optimization is performed with Group Relative Policy Optimization (GRPO), which samples a group of responses per query, normalizes their returns, and computes a group‑wise advantage. A known issue in GRPO is entropy collapse: when all sampled responses receive low reward, the advantage becomes zero and learning stalls. To counter this, the authors propose Response Mutation Operation (RMO). In each training batch, RMO randomly replaces k low‑reward responses with either the ground‑truth answer or a high‑reward reference, artificially increasing the variance of the advantage estimates. This re‑injects gradient signal, enabling the policy to escape saddle points and continue improving.

Training proceeds in two stages. Stage 1 uses only the regression component (λ₁ = λ₃ = 1, λ₂ = 0) to establish a solid baseline for predicting ordinal values. Stage 2 activates the full ranking reward (λ₁ = λ₂ = λ₃ = 1) and applies RMO, allowing the model to jointly refine regression and ranking performance.

Experiments are conducted on three benchmarks: UTKFace (facial age estimation), COCO‑REM (object counting treated as an ordinal task), and AVA (image aesthetic assessment). The underlying vision‑language model is Qwen2.5‑VL in both 3‑billion and 7‑billion parameter versions. Results show consistent gains over the base model and supervised fine‑tuning (SFT). For the 7‑B model on UTKFace, mean absolute error (MAE) drops from 4.02 to 3.81 and Kendall’s τ rises from 0.843 to 0.921. On COCO‑REM, accuracy improves from 68.73 % to 71.80 %, and on AVA the Spearman rank correlation increases from 0.783 to 0.803. Ablation studies reveal that (i) using only regression or only ranking rewards already outperforms the baseline, but the combination yields the best results, (ii) RMO contributes a measurable performance boost (e.g., MAE reduction from 4.17 to 4.02), and (iii) the two‑stage schedule stabilizes training by preventing reward conflict early on.

In conclusion, RARL demonstrates that a verifiable‑reward‑driven RL paradigm can effectively integrate regression and ranking objectives, overcoming the limitations of separate supervised pipelines. The Response Mutation Operation and staged training address common RL pitfalls such as entropy collapse and gradient vanishing, leading to robust and sample‑efficient learning. The framework achieves state‑of‑the‑art results across diverse ordinal tasks, suggesting a promising direction for future research on multi‑modal, ordinal‑aware systems, especially in settings with noisy or scarce supervision.