Diffusion Models with Heavy-Tailed Targets: Score Estimation and Sampling Guarantees

Score-based diffusion models have become a powerful framework for generative modeling, with score estimation as a central statistical bottleneck. Existing guarantees for score estimation largely focus on light-tailed targets or rely on restrictive assumptions such as compact support, which are often violated by heavy-tailed data in practice. In this work, we study conventional (Gaussian) score-based diffusion models when the target distribution is heavy-tailed and belongs to a Sobolev class with smoothness parameter $β>0$. We consider both exponential and polynomial tail decay, indexed by a tail parameter $γ$. Using kernel density estimation, we derive sharp minimax rates for score estimation, revealing a qualitative dichotomy: under exponential tails, the rate matches the light-tailed case up to polylogarithmic factors, whereas under polynomial tails the rate depends explicitly on $γ$. We further provide sampling guarantees for the associated continuous reverse dynamics. In total variation, the generated distribution converges at the minimax optimal rate $n^{-β/(2β+d)}$ under exponential tails (up to logarithmic factors), and at a $γ$-dependent rate under polynomial tails. Whether the latter sampling rate is minimax optimal remains an open question. These results characterize the statistical limits of score estimation and the resulting sampling accuracy for heavy-tailed targets, extending diffusion theory beyond the light-tailed setting.

💡 Research Summary

This paper tackles a fundamental gap in the theory of score‑based diffusion models (SGMs): the statistical analysis of score estimation and sampling when the target distribution has heavy tails. While most existing guarantees assume sub‑Gaussian or compactly supported data, many real‑world datasets—finance, imaging, and scientific measurements—exhibit polynomial or otherwise heavy‑tailed behavior. The authors consider a standard Gaussian forward diffusion (Brownian motion) and study two tail regimes for the target density p₀: (i) exponential decay (including sub‑Gaussian) parameterized by a tail exponent γ>0, and (ii) polynomial decay with tail index γ>d. The target is assumed to belong to a Sobolev class W^{β,2}(ℝᵈ) with smoothness β>0.

Methodology
The paper adopts a non‑parametric kernel approach to estimate the time‑dependent score ∇log p_t. For each diffusion time t, a kernel density estimator \hat{p}_t with bandwidth h is constructed, and the score is approximated by the gradient of its log. The authors carefully balance bias and variance, choosing h to depend on the sample size n, the smoothness β, the dimension d, and, crucially, the tail index γ. This yields two distinct convergence regimes:

• Exponential tails – The bias behaves as in the light‑tailed case, and the optimal bandwidth leads to a minimax rate for score estimation of order n^{‑β/(2β+d)} up to polylogarithmic factors. This matches known results for sub‑Gaussian targets.

• Polynomial tails – The heavy tail inflates the variance term. The optimal bandwidth scales as h≈n^{‑1/(2β+d+2γ)}, and the resulting minimax rate becomes n^{‑(γ+1)/(2γ+d+2)}. The authors prove a matching lower bound via a Le Cam/Fano construction of least‑favorable distributions, establishing that the rate is indeed minimax optimal.

Minimax Lower Bounds
For the polynomial regime, the lower bound construction explicitly incorporates the tail index γ, showing that any estimator must suffer the same γ‑dependent penalty. In the exponential regime, the lower bound reduces to the classic Sobolev result, confirming that the kernel estimator is nearly optimal (up to logarithmic terms).

Sampling Guarantees
With the estimated score \hat{s}_t, the paper studies the continuous‑time reverse SDE d\hat{Y}t = \hat{s}{T‑t}(\hat{Y}_t)dt + dW_t, initialized from the approximate Gaussian marginal \hat{p}_T = N(0,TI_d). Using Girsanov’s theorem and a variational representation of total variation (TV), the authors propagate the score estimation error through the reverse dynamics.

• In the exponential tail case, the TV distance between the generated law \mathcal{L}(\hat{Y}_0) and the true target p₀ decays at the optimal rate n^{‑β/(2β+d)} (up to polylog factors), exactly mirroring the minimax rate for score estimation.

• In the polynomial tail case, the heavy tail amplifies the error propagation. The derived TV bound is
  O!\bigl(n^{‑2β(γ+1) / \bigl(4β(d+γ+1)+d(d+2γ+2)\bigr)}\bigr).
  This rate explicitly depends on γ; as γ decreases (heavier tails), the convergence slows dramatically. Whether this bound is minimax optimal remains an open question.

Implications
The analysis demonstrates that the standard Gaussian forward diffusion remains statistically viable for heavy‑tailed data, but the price of heavy tails is a γ‑dependent slowdown in both score estimation and sampling. Practitioners working with heavy‑tailed datasets should expect to need larger sample sizes or to adjust kernel bandwidths accordingly. The results also provide a theoretical foundation for designing diffusion models that are robust to tail heaviness, potentially guiding the choice of noise schedules or alternative forward processes.

Limitations and Future Work
The paper focuses on continuous‑time reverse dynamics and does not analyze discretization errors introduced by practical solvers (Euler‑Maruyama, higher‑order schemes). Moreover, the optimality of the polynomial‑tail sampling rate is not settled; tighter lower bounds for the sampling problem are left for future research. Extending the framework to non‑Gaussian forward noises (e.g., t‑distributions, α‑stable processes) is another promising direction, as such noises may better match heavy‑tailed data but lack tractable transition densities.

Overall Contribution
By providing sharp minimax rates for score estimation under heavy‑tailed Sobolev densities and translating these rates into concrete total‑variation sampling guarantees, the paper significantly broadens the theoretical understanding of diffusion models beyond the sub‑Gaussian world. It clarifies how tail behavior quantitatively impacts statistical efficiency, offering both rigorous guarantees and practical insights for the growing community of researchers and engineers deploying diffusion models on heavy‑tailed data.