Acceleration via Perturbations on Low-resolution Ordinary Differential Equations

Recently, the high-resolution ordinary differential equation (ODE) framework, which retains higher-order terms, has been proposed to analyze gradient-based optimization algorithms. Through this framework, the term $\nabla^2 f(X_t)\dot{X_t}$, known as the gradient-correction term, was found to be essential for reducing oscillations and accelerating the convergence rate of function values. Despite the importance of this term, simply adding it to the low-resolution ODE may sometimes lead to a slower convergence rate. To fully understand this phenomenon, we propose a generalized perturbed ODE and analyze the role of the gradient and gradient-correction perturbation terms under both continuous-time and discrete-time settings. We demonstrate that while the gradient-correction perturbation is essential for obtaining accelerations, it can hinder the convergence rate of function values in certain cases. However, this adverse effect can be mitigated by involving an additional gradient perturbation term. Moreover, by conducting a comprehensive analysis, we derive proper choices of perturbation parameters. Numerical experiments are also provided to validate our theoretical findings.

💡 Research Summary

The paper investigates a puzzling phenomenon observed when the gradient‑correction term ∇²f(Xₜ)·Ẋₜ, which is crucial in high‑resolution ODE analyses of accelerated first‑order methods, is added directly to the low‑resolution ODE that models Nesterov’s accelerated gradient (NAG‑SC) and the Heavy‑Ball (HB) method. While the high‑resolution ODEs (e.g., ¨Xₜ+2√μ·Ẋₜ+(1+√μ s)∇f(Xₜ)+√s ∇²f(Xₜ)·Ẋₜ=0) faithfully capture the discrete dynamics and yield the optimal exponential decay O(e^{-√μ t}), the naive low‑resolution counterpart with the same correction term (e.g., ¨Xₜ+2√μ·Ẋₜ+β∇²f(Xₜ)·Ẋₜ+∇f(Xₜ)=0) can actually slow down the convergence of the function value to O(e^{-√μ t/2}).

To explain and remedy this, the authors propose a generalized perturbed ODE:

  ¨Xₜ+2√μ·Ẋₜ+(1+Δ₁)∇f(Xₜ)+Δ₂∇²f(Xₜ)·Ẋₜ=0  (1.5)

where Δ₁, Δ₂ ≥ 0 are tunable perturbation parameters. They interpret Δ₁∇f as a “restoring force” that speeds the particle back to the optimum but may increase oscillations, while Δ₂∇²f·Ẋₜ acts as a velocity‑dependent damping that suppresses oscillations but can decelerate the approach to the optimum if used alone.

Using a Lyapunov function

 E(t)=e^{√μ t}