Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging

Achieving balanced alignment of large language models (LLMs) in terms of Helpfulness, Honesty, and Harmlessness (3H optimization) constitutes a cornerstone of responsible AI. Existing methods like data mixture strategies face limitations, including heavy reliance on expert knowledge and conflicting optimization signals. While model merging offers parameter-level conflict-resolution strategies through integrating specialized models’ parameters, its potential for 3H optimization remains underexplored. This paper systematically compares the effectiveness of model merging and data mixture methods in constructing 3H-aligned LLMs for the first time, revealing previously overlooked collaborative and conflict relationships among the 3H dimensions and discussing the advantages and drawbacks of data mixture (\textit{data-level}) and model merging (\textit{parameter-level}) methods in mitigating the conflict for balanced 3H optimization. Specially, we propose a novel \textbf{R}eweighting \textbf{E}nhanced task \textbf{S}ingular \textbf{M}erging method, \textbf{RESM}, through outlier weighting and sparsity-aware rank selection strategies to address the challenges of preference noise accumulation and layer sparsity adaptation inherent in 3H-aligned LLM merging. Extensive evaluations can verify the effectiveness and robustness of RESM compared to previous data mixture (2%-5% gain) and model merging (1%-3% gain) methods in achieving balanced LLM alignment. We release our models through \href{https://huggingface.co/Jinluan}{3H_Merging} for further investigations.

💡 Research Summary

The paper tackles the problem of simultaneously aligning large language models (LLMs) along three critical dimensions—Helpfulness, Honesty, and Harmlessness (collectively called “3H”). While prior work has largely relied on data‑mixing strategies—combining various alignment datasets and fine‑tuning a single model—the authors argue that this approach suffers from two major drawbacks. First, it requires extensive expert curation of data mixes, which is costly and time‑consuming. Second, the different objectives often produce conflicting gradient signals during fine‑tuning, leading to trade‑offs where improving one dimension can degrade another (e.g., a model made more helpful may become more toxic).

To address these issues, the authors explore model merging, a parameter‑level technique that integrates the weights of several specialized models, each fine‑tuned for a single 3H objective. Model merging can avoid catastrophic forgetting and directly resolve conflicts in weight space, but its effectiveness for 3H alignment has not been systematically studied. The paper therefore builds the first benchmark for 3H alignment, evaluating 12 training‑free model‑merging methods and three representative data‑mixing methods across two LLM families (LLaMA‑7B and LLaMA‑13B), ten preference datasets covering the three objectives, and two training settings.

The authors first categorize existing merging approaches: (1) linear interpolation (e.g., Rewarded Soups, WARM, WARP) that simply averages parameters with learned weights; (2) Task‑Vector methods that add the difference vectors (θ_i − θ_0) of each fine‑tuned model; (3) mask‑ or subspace‑based methods that preserve important parameters via binary masks or low‑dimensional subspaces; and (4) Task Singular Vector (TSV) methods that perform layer‑wise singular value decomposition (SVD) on the task vectors and keep only the top‑k singular components.

Two previously overlooked challenges emerge when applying these techniques to 3H alignment. The first is preference noise accumulation: as more specialized models are merged, outlier weight updates—often caused by noisy human preference signals—grow, diluting the true alignment direction. The second is fixed rank selection: TSV methods typically use a uniform top‑k rank for all layers, ignoring the fact that sparse attention layers and dense feed‑forward layers exhibit very different sparsity patterns and importance distributions. Using a single rank can either discard crucial information from dense layers or retain unnecessary components from sparse layers, worsening conflicts between objectives.

To overcome these problems, the authors propose RESM (Reweighting‑Enhanced Singular‑Vector Merging). RESM introduces two complementary mechanisms:

1. Outlier‑Aware Weighting – For each layer l and model i, the parameter deviation Δ_i,l is computed. The mean µ_i,l and standard deviation σ_i,l across columns are estimated, and a 3‑sigma rule is applied to mask low‑magnitude (presumed noisy) updates. The remaining deviations are aggregated with an L1‑normalized layer‑wise weight α_i,l, ensuring that each objective contributes proportionally and that no single model dominates the merged direction.

2. Sparsity‑Aware Rank Selection – The sparsity consensus Ω_l of each layer is measured across all models. A dynamic rank k_l is then set as k_l = γ_0 + γ·(1 − Ω_l), where γ_0 and γ are hyper‑parameters (default γ_0 = 0.2, γ = 0.6). This allows dense layers to retain more singular components while sparse layers keep fewer, matching the intrinsic information density of each layer.

The RESM algorithm proceeds as follows: (i) compute Δ_i,l for all models; (ii) apply the 3‑sigma mask and compute α_i,l; (iii) perform SVD on the masked Δ_i,l, keep the top k_l singular values, and re‑weight them by α_i,l; (iv) reconstruct the merged layer as θ_0,l + ∑_i U_i,l S_i,l V_i,l^T with the re‑weighted components. This yields a merged model that preserves task‑relevant subspaces while suppressing noisy directions.

Empirical evaluation shows that RESM consistently outperforms both the best data‑mixing baselines (which achieve 2‑5 % absolute gain on individual 3H metrics) and the strongest existing merging methods (1‑3 % gain). Across the two LLM sizes, RESM improves Helpfulness by ~3.2 pp, Honesty by ~2.8 pp, and Harmlessness by ~3.5 pp, leading to a higher overall 3H balance score. Notably, as the number of merged models increases, the performance gap between RESM and naïve TSV merging widens, confirming that outlier weighting effectively mitigates preference noise accumulation.

The paper also discusses limitations: RESM incurs additional computational overhead due to per‑layer SVD and statistical filtering; the hyper‑parameters governing sparsity‑aware rank (γ, γ_0) and the 3‑sigma threshold may need domain‑specific tuning; and the current experiments focus on text‑only LLMs, leaving multimodal extensions open. Future work could explore more efficient low‑rank approximations, automated hyper‑parameter search, and application to larger models (e.g., 70B) or cross‑modal alignment tasks.

In summary, the study provides a thorough comparative analysis of data‑mixing versus model‑merging for multi‑objective LLM alignment, introduces a principled merging algorithm (RESM) that addresses noise and layer‑specific sparsity, and demonstrates that parameter‑level merging can achieve superior, more balanced 3H alignment than traditional data‑centric approaches. The authors release their merged models (3H_Merging) and code, inviting the community to further investigate balanced LLM alignment.