Saddle Point Evasion via Curvature-Regularized Gradient Dynamics

Nonconvex optimization underlies many modern machine learning and control tasks, where saddle points pose the dominant obstacle to reliable convergence in high-dimensional settings. Escaping these saddle points deterministically and at a controllable rate remains an open challenge: gradient descent is blind to curvature, stochastic perturbation methods lack deterministic guarantees, and Newton-type approaches suffer from Hessian singularity. We present Curvature-Regularized Gradient Dynamics (CRGD), which augments the objective with a smooth penalty on the most negative Hessian eigenvalue, yielding an augmented cost that serves as an optimization Lyapunov function with user-selectable convergence rates to second-order stationary points. Numerical experiments on a nonconvex matrix factorization example confirm that CRGD escapes saddle points across all tested configurations, with escape time that decreases with the eigenvalue gap, in contrast to gradient descent, whose escape time grows inversely with the gap.

💡 Research Summary

The paper addresses a fundamental obstacle in high‑dimensional non‑convex optimization: strict saddle points that impede convergence of first‑order methods. Classical gradient descent (GD) cannot distinguish saddles from minima and, in the worst case, requires a number of iterations proportional to the inverse of the smallest negative Hessian eigenvalue (δ⁻¹). Stochastic perturbations (e.g., perturbed GD) provide probabilistic escape guarantees but lack deterministic performance guarantees. Newton‑type approaches and cubic regularization exploit second‑order information to accelerate escape, yet they suffer from Hessian singularities, require expensive Hessian‑vector products, and do not allow the practitioner to prescribe a desired convergence rate.

To overcome these limitations, the authors adopt the Optimization Lyapunov Function (OLF) framework, which treats the optimizer as a feedback controller for a single‑integrator system ˙x = u. A Lyapunov candidate V(x) ≥ 0 is chosen so that its zero set coincides with the set of second‑order stationary points (SOSP) X*. The user also selects a decay law σ(V, t) (exponential, finite‑time, fixed‑time, or prescribed‑time). The feedback law u = –σ(V, t) ∇V / ‖∇V‖² then guarantees exact Lyapunov decay ˙V = –σ(V, t), thereby enforcing the chosen convergence profile.

The key innovation is the construction of a curvature‑aware Lyapunov function. The authors augment the original objective J(x) with a smooth penalty on the most negative Hessian eigenvalue λ_min(x): \