Learning to summarize user information for personalized reinforcement learning from human feedback

As everyday use cases of large language model (LLM) AI assistants have expanded, it is becoming increasingly important to personalize responses to align to different users’ preferences and goals. While reinforcement learning from human feedback (RLHF) is effective at improving LLMs to be generally more helpful and fluent, it does not account for variability across users, as it models the entire user population with a single reward model, meaning it assumes that everyone’s preferences are the same. We present a novel framework, Preference Learning Using Summarization (PLUS), that uses reinforcement learning (RL) to learn to produce text-based summaries of each user’s preferences, characteristics, and past conversations. These summaries condition the reward model, enabling it to make personalized predictions about the types of responses valued by each user. Both the user-summarization model and reward model are trained simultaneously, creating an online co-adaptation loop. We show that in contrast to the standard Bradley-Terry model, summaries produced by PLUS capture diverse aspects of user preferences, achieving a 11-77/% improvement in reward model accuracy. Key strengths of PLUS are: (1) robust performance with new users and conversation topics, achieving a 25% improvement over the best personalized reward model technique used for RLHF; (2) zero-shot personalization with state-of-the-art proprietary models like GPT-4 (e.g., PLUS-summary-conditioned responses achieved a 72% win rate compared to 28% for default GPT-4o); (3) learning from flexible user contexts beyond preference labels, and (4) interpretable representation of users, enabling greater transparency and user control in pluralistic LLM alignment.

💡 Research Summary

The paper addresses a fundamental limitation of current Reinforcement Learning from Human Feedback (RLHF) pipelines: they rely on a single reward model, typically instantiated as a Bradley‑Terry‑Luce (BTL) pairwise preference predictor, which assumes that all users share the same utility function. In practice, users exhibit highly heterogeneous and sometimes conflicting preferences, especially as large language model (LLM) assistants become ubiquitous across cultures, domains, and personal goals. To overcome this “pluralistic alignment” gap, the authors propose Preference Learning Using Summarization (PLUS), a novel framework that learns a textual summary of each user’s preferences, characteristics, and conversation history, and conditions the reward model on these summaries.

Core Architecture
PLUS consists of two tightly coupled components:

1. Summarizer (πθ) – a language model that, given a user context c (e.g., past dialogues, explicit or implicit feedback), generates a concise natural‑language summary z. Rather than training this module with supervised labels, the authors treat summary generation as a reinforcement‑learning problem. The reward signal is derived from the downstream reward model’s negative log‑likelihood loss on preference pairs when conditioned on the sampled summary. PPO (Proximal Policy Optimization) with Generalized Advantage Estimation (GAE) is used to update πθ at the token level, allowing the policy to learn which textual cues most improve downstream preference prediction.
2. Summary‑conditioned Reward Model (rϕ) – a standard BTL‑style pairwise predictor that now takes the generated summary as an additional conditioning variable: p(sA ≻ sB | z) = σ(rϕ(sA|z) – rϕ(sB|z)). By feeding a natural‑language description rather than a fixed embedding, the model can exploit the full expressive power of LLMs to reason about nuanced user traits (e.g., “prefers concise answers”, “values scientific citations”, “avoids political topics”).

Training proceeds in an alternating co‑adaptation loop: (a) freeze the summarizer, sample summaries for each context, and train rϕ to minimize the negative log‑likelihood across all preference pairs; (b) freeze rϕ, compute the per‑summary loss, and update πθ with PPO to maximize the expected reward (i.e., minimize the loss). This loop continues until convergence, effectively aligning the summarizer’s output distribution with the reward model’s needs.

Key Empirical Findings

• Accuracy Gains: Compared to a vanilla BTL model, PLUS improves reward‑model accuracy by 11 %–77 % across several benchmarks (Pets, UltraFeedback).
• Generalization: When evaluated on unseen users and new conversation topics, PLUS achieves a 25 % relative improvement over the strongest existing personalized RLHF baseline.
• Zero‑Shot Personalization of Proprietary Models: By feeding PLUS‑generated summaries to GPT‑4 (without any fine‑tuning), the authors obtain a 72 % win‑rate in pairwise human evaluations, versus 28 % for the default GPT‑4o. This demonstrates that the approach can personalize even closed‑source, high‑capacity models.
• Scalability to Real‑World Heterogeneous Data: The method is evaluated on PRISM, a large‑scale pluralistic dataset comprising 1,500 users from 75 countries interacting with 20 LLMs (≈9,000 conversations). PLUS is among the first to report reward‑model results on PRISM, showing substantial performance lifts, confirming its robustness to cultural and linguistic diversity.
• Interpretability & User Control: Summaries are human‑readable. Users can inspect, edit, or overwrite them, directly influencing the downstream reward model. This contrasts with embedding‑based user vectors that are opaque and difficult to modify.

Comparisons to Prior Work
Previous personalized RLHF approaches (e.g., Variational Preference Learning, Distributional Preference Learning, LoRA‑based adapters) encode users as low‑dimensional vectors or linear weights. While computationally efficient, these representations sacrifice interpretability and may discard subtle textual cues. PLUS leverages the generative capacity of LLMs to keep the user representation in natural language, preserving rich semantic information. The authors also benchmark against a naïve “prompt‑only” summarization (using GPT‑4 to generate summaries without fine‑tuning) and a “full‑history conditioning” baseline; both underperform relative to the learned, concise summaries, especially under topic shift.

Limitations & Future Directions

• Computational Overhead: Generating and storing summaries for every interaction incurs higher latency and memory costs than static embeddings.
• Training Instability: Early in training, low‑quality summaries can provide noisy rewards, potentially destabilizing PPO updates. The authors mitigate this with a strong pre‑trained LLM initialization and length constraints, but further curriculum or reward‑shaping strategies may be needed for larger models.
• Privacy Concerns: Summaries may contain personally identifiable information; integrating differential privacy or on‑device summarization could be necessary for real‑world deployment.
• Scalability to Real‑Time Systems: While the paper demonstrates offline evaluation, integrating PLUS into a live chat service would require efficient caching and possibly distillation of the summarizer.

Overall Assessment
PLUS introduces a compelling paradigm shift: treating user personalization as a language generation problem rather than a vector embedding problem. By jointly optimizing a summarizer and a summary‑conditioned reward model via PPO, the framework achieves notable gains in accuracy, generalization, and interpretability. Its success on the PRISM dataset and zero‑shot personalization of GPT‑4 suggests strong practical relevance. Future work that addresses computational efficiency, privacy, and real‑time integration will be crucial for broader adoption, but the paper already sets a solid foundation for pluralistic, transparent alignment of LLM assistants.