Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness

This work introduces a hybrid non-Euclidean optimization method which generalizes gradient norm clipping by combining steepest descent and conditional gradient approaches. The method achieves the best of both worlds by establishing a descent property under a generalized notion of ($L_0$,$L_1$)-smoothness. Weight decay is incorporated in a principled manner by identifying a connection to the Frank-Wolfe short step. In the stochastic case, we show an order optimal $O(n^{-1/4})$ convergence rate by leveraging a momentum based gradient estimator. We discuss how to instantiate the algorithms for deep learning, which we dub Clipped Scion, and demonstrate their properties on image classification and language modeling. The code is available at https://github.com/LIONS-EPFL/ClippedScion.

💡 Research Summary

This paper introduces a novel hybrid optimization framework that unifies steepest descent (SD) and unconditional conditional gradient (uCG) methods under a non‑Euclidean setting. The authors observe that while SD guarantees descent it requires a small stepsize (γ < 2/L), whereas uCG can take arbitrarily large steps but lacks a descent guarantee, especially near critical points. To obtain the “best of both worlds,” they propose Generalized Gradient Norm Clipping (GGNC), which applies a trust‑region of radius ρ in an arbitrary norm and automatically switches between a normalized uCG step when the gradient norm is large and a standard SD step when the norm is small.

A central technical contribution is the exploitation of the dual relationship between the sharp‑operator (the maximizer of ⟨d,x⟩ – ½‖x‖²) and the linear minimization oracle (lmo) that solves min_{x∈D}⟨d,x⟩ for a norm‑ball D. This relationship, d♯ = –‖d‖* lmo(d), allows the authors to express GGNC either via the sharp‑operator or via lmo, making the algorithm applicable to any norm for which an lmo is available (e.g., ℓ∞, spectral, product/max‑norm).

The paper extends the recently studied (L₀,L₁)‑smoothness condition—originally defined for Euclidean spaces—to arbitrary norms. Under this generalized smoothness, they prove that GGNC enjoys a true descent property for a fixed stepsize γ, unlike uCG. Moreover, they show that GGNC inherits the fast O(1/k) decay of the gradient norm from uCG in the early phase (large gradients) and the exact convergence of SD in the later phase (small gradients).

In the stochastic setting, the authors introduce a momentum‑based gradient estimator d_k = α_k∇f(x_k,ξ_k)+(1–α_k)d_{k–1} to mitigate bias introduced by the non‑linearity of lmo. Using this estimator, they derive an order‑optimal convergence rate of O(n^{–1/4}) for the expected suboptimality, matching known lower bounds for non‑convex stochastic optimization under similar smoothness assumptions.

Weight decay is incorporated by recognizing that applying a short‑step Frank‑Wolfe update with a radius β is equivalent to a clipped step with radius ρ = βγ. This insight yields a principled way to combine clipping with L2 regularization while preserving convergence guarantees.

The authors instantiate GGNC for deep neural networks via a product norm (max‑norm across layers), leading to the “Clipped Scion” algorithms (unconstrained and constrained variants). Specific cases include:

• Clipped Sign (ℓ∞ norm) – updates use sign(d_k) with a scalar step.
• Clipped Spectral (Schatten‑∞ norm) – updates use the top singular vectors of d_k.
• Clipped Scion – layer‑wise max‑norm clipping, analogous to LARS but with a global dual‑norm scaling.

Empirical evaluations on CIFAR‑10/100 image classification and large‑scale language modeling (Transformer‑based) demonstrate that Clipped Scion converges faster in the early epochs due to large steps, and stabilizes later thanks to automatic clipping. Compared to standard gradient clipping, sign‑clipping, and spectral‑clipping baselines, it achieves modest but consistent improvements (≈1–2 % higher accuracy or lower perplexity) and exhibits better robustness to learning‑rate schedules.

Overall, the paper provides a rigorous theoretical foundation for non‑Euclidean gradient clipping, bridges it with Frank‑Wolfe short‑step methods, and delivers practical algorithms that improve training stability and performance in modern deep learning workloads. Future work may explore adaptive ρ schedules, more complex composite norms, and distributed implementations of the required lmo subroutines.