On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models

Entropy serves as a critical metric for measuring the diversity of outputs generated by large language models (LLMs), providing valuable insights into their exploration capabilities. While recent studies increasingly focus on monitoring and adjusting entropy to better balance exploration and exploitation in reinforcement fine-tuning (RFT), a principled understanding of entropy dynamics during this process is yet to be thoroughly investigated. In this paper, we establish a theoretical framework for analyzing the entropy dynamics during the RFT process, which begins with a discriminant expression that quantifies entropy change under a single logit update. This foundation enables the derivation of a first-order expression for entropy change, which can be further extended to the update formula of Group Relative Policy Optimization (GRPO). The corollaries and insights drawn from the theoretical analysis inspire the design of entropy control methods, and also offer a unified lens for interpreting various entropy-based methods in existing studies. We provide empirical evidence to support the main conclusions of our analysis and demonstrate the effectiveness of the derived entropy-discriminator clipping methods. This study yields novel insights into RFT training dynamics, providing theoretical support and practical strategies for optimizing the exploration-exploitation balance during LLM fine-tuning.

💡 Research Summary

The paper tackles a fundamental gap in the understanding of how entropy—used as a proxy for output diversity—evolves during reinforcement fine‑tuning (RFT) of large language models (LLMs). While recent works have begun to monitor or manipulate entropy to balance exploration and exploitation, they largely rely on heuristics without a principled theoretical basis. This work introduces a rigorous analytical framework that starts from the simplest possible operation—a single logit perturbation—and builds up to the full Group Relative Policy Optimization (GRPO) update used in state‑of‑the‑art RFT pipelines.

Single‑logit dynamics.
The authors model a small change ε to the logit of token k (δz = ε e_k). Using the Jacobian of the softmax, they derive exact first‑order changes in the probability distribution: δp_k = ε p_k(1 − p_k) and δp_i = −ε p_i p_k for all i ≠ k. This shows that increasing a token’s logit redistributes probability mass uniformly from all other tokens, and vice‑versa.

Entropy discriminator S*.
Plugging the probability changes into a first‑order Taylor expansion of the Shannon entropy yields ΔH = −ε S* + O(ε²), where S* = p_k(H + log p_k). S* captures the relationship between a token’s probability p_k and the overall entropy H of the distribution. Its sign is determined by whether p_k is larger or smaller than e^{−H}. Consequently, rewarding a low‑probability token (ε > 0, p_k < e^{−H}) raises entropy, while rewarding a high‑probability token reduces entropy. Penalizing a token reverses the effect. This simple formula explains the empirically observed “entropy collapse” when models are repeatedly rewarded for safe, high‑probability responses.

Extension to GRPO.
GRPO updates all tokens in a sampled response simultaneously, scaling the gradient by α = η r A (learning rate η, importance ratio r, advantage A). The authors show that the first‑order entropy change becomes ΔH = −α (S* − E_p