Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignment

Group Relative Policy Optimization (GRPO) has proven highly effective in enhancing the alignment capabilities of Large Language Models (LLMs). However, current adaptations of GRPO for the flow matching-based image generation neglect a foundational conflict between its core principles and the distinct dynamics of the visual synthesis process. This mismatch leads to two key limitations: (i) Uniformly applying a sparse terminal reward across all timesteps impairs temporal credit assignment, ignoring the differing criticality of generation phases from early structure formation to late-stage tuning. (ii) Exclusive reliance on relative, intra-group rewards causes the optimization signal to fade as training converges, leading to the optimization stagnation when reward diversity is entirely depleted. To address these limitations, we propose Value-Anchored Group Policy Optimization (VGPO), a framework that redefines value estimation across both temporal and group dimensions. Specifically, VGPO transforms the sparse terminal reward into dense, process-aware value estimates, enabling precise credit assignment by modeling the expected cumulative reward at each generative stage. Furthermore, VGPO replaces standard group normalization with a novel process enhanced by absolute values to maintain a stable optimization signal even as reward diversity declines. Extensive experiments on three benchmarks demonstrate that VGPO achieves state-of-the-art image quality while simultaneously improving task-specific accuracy, effectively mitigating reward hacking. Project webpage: https://yawen-shao.github.io/VGPO/.

💡 Research Summary

This paper, “Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignment,” identifies critical limitations in directly applying Group Relative Policy Optimization (GRPO)—a successful reinforcement learning from human feedback (RLHF) technique for large language models—to flow matching-based image generation models. It proposes a novel framework, Value-Anchored Group Policy Optimization (VGPO), to overcome these issues.

The authors argue that current adaptations like Flow-GRPO suffer from two fundamental mismatches. First, there is a temporal misalignment: GRPO distributes a single, sparse terminal reward (based on the final image) uniformly across all denoising timesteps. This ignores the varying criticality of different generation phases, from crucial early-stage structural decisions to late-stage refinements, leading to faulty credit assignment and inefficient learning. Second, there is an over-reliance on reward diversity: GRPO’s optimization signal derives from the relative differences in rewards within a sampled group. As training converges and the model produces consistently high-quality images, reward variance diminishes, causing the optimization gradient to vanish and leading to policy stagnation.

VGPO addresses these challenges through two synergistic core components:

1. Temporal Cumulative Reward Mechanism (TCRM): This mechanism transforms the sparse terminal reward into dense, process-aware value estimates. It defines an instant reward for each generation step by performing a one-step deterministic ODE sampling after taking an action (a denoising step). This produces a proxy for the immediate outcome, which is evaluated by the reward model. To avoid myopic optimization, TCRM then estimates the long-term cumulative discounted reward (Q-value) for each action using Monte-Carlo estimation over the sampled trajectory. These Q-values represent the forward-looking worth of each denoising decision and are used to re-weight the importance of each timestep during policy updates, prioritizing more critical stages of generation.

2. Adaptive Dual Advantage Estimation (ADAE): This component tackles the optimization stagnation problem. It replaces GRPO’s standard group normalization with a novel advantage computation that adaptively fuses relative advantage (based on group variance) and absolute reward value. The core innovation is an adaptive mechanism that automatically shifts the optimization focus. When reward diversity within a group is high, it leverages relative comparisons. As diversity diminishes and approaches zero, it seamlessly transitions to optimizing based on absolute reward values. The paper provides a theoretical proof that ADAE ensures a persistent optimization signal even when reward variance is fully depleted, preventing policy collapse.

The authors conduct extensive experiments across three challenging benchmarks: compositional image generation, visual text rendering, and human preference alignment. Results demonstrate that VGPO achieves state-of-the-art performance, simultaneously improving both general image quality metrics (e.g., lower FID, higher CLIP Score) and task-specific accuracy (e.g., text correctness in generated images). Crucially, it achieves this while effectively mitigating reward hacking, indicating a genuine alignment between the reward signal and human-perceived quality. In summary, VGPO provides a principled reframing of value estimation across both the temporal dimension of the generative process and the group dimension of policy optimization, offering a robust and effective framework for aligning flow-based generative models.