Towards The Implicit Bias on Multiclass Separable Data Under Norm Constraints

Implicit bias induced by gradient-based algorithms is essential to the generalization of overparameterized models, yet its mechanisms can be subtle. This work leverages the Normalized Steepest Descent} (NSD) framework to investigate how optimization geometry shapes solutions on multiclass separable data. We introduce NucGD, a geometry-aware optimizer designed to enforce low rank structures through nuclear norm constraints. Beyond the algorithm itself, we connect NucGD with emerging low-rank projection methods, providing a unified perspective. To enable scalable training, we derive an efficient SVD-free update rule via asynchronous power iteration. Furthermore, we empirically dissect the impact of stochastic optimization dynamics, characterizing how varying levels of gradient noise induced by mini-batch sampling and momentum modulate the convergence toward the expected maximum margin solutions.Our code is accessible at: https://github.com/Tsokarsic/observing-the-implicit-bias-on-multiclass-seperable-data.

💡 Research Summary

This paper investigates the implicit bias of gradient‑based optimization algorithms on multiclass linearly separable data through the lens of the Normalized Steepest Descent (NSD) framework. The authors first review prior work showing that plain gradient descent on exponential loss converges to the ℓ₂‑max‑margin SVM solution, and that this result extends to the multiclass setting. They then point out that modern deep‑learning practice relies on adaptive methods such as Adam, SignGD, and Muon, each of which has been shown to implicitly favor solutions defined by different norms (ℓ∞, spectral, etc.). To unify these disparate observations, the paper adopts the NSD framework, which models each update as a steepest‑descent step constrained to a ball defined by a chosen norm. Within this framework, the direction of the update maximizes the inner product with the current momentum (or gradient) while staying inside the norm ball.

The core contribution is the introduction of NucGD, a geometry‑aware optimizer derived from a nuclear‑norm (trace‑norm) constraint. The nuclear norm, being the ℓ₁ sum of singular values, serves as the tightest convex surrogate for matrix rank. By solving the constrained maximization problem analytically (Theorem 2), the authors show that the optimal update direction is γ u₁v₁ᵀ, where u₁ and v₁ are the leading left and right singular vectors of the momentum matrix M. This direction simultaneously satisfies the nuclear‑norm ball constraint and yields the maximal inner product, thereby defining the NucGD update rule.

Because computing a full singular value decomposition (SVD) at every iteration is computationally prohibitive for large‑scale problems, the authors propose an SVD‑free implementation based on asynchronous power iteration. They observe that the leading singular vector of M is also the dominant eigenvector of N = M Mᵀ. By initializing the power iteration with the estimate from the previous step and performing only a single iteration per training step, they obtain a highly accurate approximation of u₁v₁ᵀ at a cost comparable to a matrix‑vector multiplication. The resulting Algorithm 2 retains the theoretical guarantees of NucGD while scaling to high‑dimensional settings.

Theoretical analysis extends the existing NSD convergence result (Theorem 1) to the nuclear‑norm case. Under standard assumptions (linearly separable data, diminishing step size of order 1/t, and momentum parameter μ ∈