FedRD: Reducing Divergences for Generalized Federated Learning via Heterogeneity-aware Parameter Guidance

Heterogeneous federated learning (HFL) aims to ensure effective and privacy-preserving collaboration among different entities. As newly joined clients require significant adjustments and additional training to align with the existing system, the problem of generalizing federated learning models to unseen clients under heterogeneous data has become progressively crucial. Consequently, we highlight two unsolved challenging issues in federated domain generalization: Optimization Divergence and Performance Divergence. To tackle the above challenges, we propose FedRD, a novel heterogeneity-aware federated learning algorithm that collaboratively utilizes parameter-guided global generalization aggregation and local debiased classification to reduce divergences, aiming to obtain an optimal global model for participating and unseen clients. Extensive experiments on public multi-domain datasets demonstrate that our approach exhibits a substantial performance advantage over competing baselines in addressing this specific problem.

💡 Research Summary

Federated learning (FL) has become a cornerstone for privacy‑preserving collaborative AI, yet its performance degrades sharply when client data are heterogeneous, i.e., when both domain shift and label shift coexist. Moreover, a model trained on participating clients often fails to generalize to newly‑joined or unseen clients whose data distribution differs from any training domain. The authors identify two previously under‑explored problems that hinder such “generalized federated learning”: (1) Optimization Divergence, where the gradient directions that reduce domain‑specific variance conflict with those that mitigate label imbalance, leading to contradictory updates; and (2) Performance Divergence, where the convergence speed and final accuracy of local models differ markedly from the global model, causing the global aggregation to be biased toward sub‑optimal client updates.

To address both issues simultaneously, the paper proposes FedRD (Federated Reduction of Divergences). The method consists of three tightly coupled components:

1. Feature Extractor and Debiased Classifier Architecture – Each client model is split into a feature extractor (f^{\phi}_i) and a classifier (f^{\psi}_i). The extractor captures domain‑specific representations, while the classifier is explicitly designed to counteract class imbalance.

2. Parameter‑Guided Domain Discrepancy Measurement – The authors treat the weights of a designated “domain‑knowledge” layer (typically a lower convolutional block) as a compact encoding of domain information. For client (i), the Euclidean distance between its extractor weights (w^{\phi}_i) and the current global extractor weights (w^{\phi}_g) is computed as
   \