On DeepSeekMoE: Statistical Benefits of Shared Experts and Normalized Sigmoid Gating

Mixture of experts (MoE) methods are a key component in most large language model architectures, including the recent series of DeepSeek models. Compared to other MoE implementations, DeepSeekMoE stands out because of two unique features: the deployment of a shared expert strategy and of the normalized sigmoid gating mechanism. Despite the prominent role of DeepSeekMoE in the success of the DeepSeek series of models, there have been only a few attempts to justify theoretically the value of the shared expert strategy, while its normalized sigmoid gating has remained unexplored. To bridge this gap, we undertake a comprehensive theoretical study of these two features of DeepSeekMoE from a statistical perspective. We perform a convergence analysis of the expert estimation task to highlight the gains in sample efficiency for both the shared expert strategy and the normalized sigmoid gating, offering useful insights into the design of expert and gating structures. To verify empirically our theoretical findings, we carry out several experiments on both synthetic data and real-world datasets for (vision) language modeling tasks. Finally, we conduct an extensive empirical analysis of the router behaviors, ranging from router saturation, router change rate, to expert utilization.

💡 Research Summary

The paper provides a thorough statistical analysis of two distinctive components of the DeepSeekMoE architecture, which is a sparsely activated mixture‑of‑experts (MoE) layer used in the recent DeepSeek series of large language models. The first component is the “shared expert” strategy, where a subset of experts is always active for every input, thereby learning common knowledge across all domains. The second component is the “normalized sigmoid gating” introduced in DeepSeek‑V3, which replaces the traditional softmax gating with a per‑expert sigmoid followed by a global normalization.

The authors formalize the problem as maximum‑likelihood estimation (MLE) of a Gaussian MoE model with two groups of experts: shared experts (indexed by k₁) and routed experts (indexed by k₂). They impose mild assumptions on compactness of the parameter space, bounded inputs, and a zero‑mean condition on the last gating parameters to avoid translation non‑identifiability. A central technical contribution is the definition of “strong identifiability” for the expert functions h₁(x,κ) and h₂(x,η). This condition requires that first‑ and second‑order partial derivatives with respect to the expert parameters are linearly independent as functions of the input x. The authors show that common feed‑forward networks using GELU, sigmoid, or tanh satisfy this condition, while linear experts do not.

Under strong identifiability, the MLE density estimator converges to the true conditional density in total variation distance at a near‑parametric rate O(√(log n / n)). By constructing a suitable loss on the mixing measures, this rate translates directly into parameter‑wise convergence. Crucially, because shared experts are always active, their parameters enjoy a fast convergence rate of order n⁻¹⁄⁴, independent of the number of routed experts. In contrast, routed experts under the traditional softmax gating have convergence rates that depend on the solvability of a system of polynomial equations; when these systems are ill‑conditioned, the rate can degrade dramatically.

The normalized sigmoid gating eliminates the polynomial‑system dependence for routed experts. By applying an independent sigmoid to each expert’s raw gate score and then normalizing across all experts, the gating function becomes a composition of smooth, monotone maps that do not create the same algebraic constraints as softmax. Consequently, routed experts achieve a convergence rate of order n⁻¹⁄², which is substantially faster than the softmax case and comparable to the shared‑expert rate. The shared‑expert convergence remains unchanged because it does not rely on the gating function.

To validate the theory, the authors conduct three sets of experiments. First, synthetic data experiments confirm that the empirical error curves follow the predicted n⁻¹⁄⁴ (shared) and n⁻¹⁄² (routed, sigmoid) rates, while softmax‑gated routed experts lag behind. Second, they embed DeepSeek‑V2 (softmax gating) and DeepSeek‑V3 (normalized sigmoid gating) MoE layers into large language‑model and vision‑language tasks. Results show that DeepSeek‑V3 consistently attains higher validation accuracy with fewer training steps, and the router exhibits reduced saturation (fewer experts become permanently dominant) and higher change rates, indicating more balanced expert utilization. Third, a detailed router analysis quantifies expert utilization, saturation, and change dynamics, revealing that shared experts occupy roughly 30 % of total traffic while routed experts are more evenly distributed under sigmoid gating.

The paper’s contributions are threefold: (1) a rigorous sample‑complexity analysis showing that shared experts require fewer data points to reach a given statistical accuracy; (2) a demonstration that normalized sigmoid gating dramatically improves the sample efficiency of routed experts compared with softmax gating; (3) extensive empirical evidence supporting the theoretical claims across synthetic and real‑world tasks. Limitations include the focus on Gaussian MoE models, the reliance on strong identifiability (which may not hold for all network architectures), and the lack of exploration of other gating normalizations or multi‑stage routing. Nonetheless, the work provides the first solid statistical foundation for the design choices in DeepSeekMoE and offers practical guidance for building more sample‑efficient, scalable MoE layers in future large‑scale models.