Step-Size Stability in Stochastic Optimization: A Theoretical Perspective

We present a theoretical analysis of stochastic optimization methods in terms of their sensitivity with respect to the step size. We identify a key quantity that, for each method, describes how the performance degrades as the step size becomes too large. For convex problems, we show that this quantity directly impacts the suboptimality bound of the method. Most importantly, our analysis provides direct theoretical evidence that adaptive step-size methods, such as SPS or NGN, are more robust than SGD. This allows us to quantify the advantage of these adaptive methods beyond empirical evaluation. Finally, we show through experiments that our theoretical bound qualitatively mirrors the actual performance as a function of the step size, even for nonconvex problems.

💡 Research Summary

The paper provides a rigorous theoretical framework for understanding how stochastic optimization algorithms react to changes in the step size (learning rate). The authors introduce a “stability index” ( \delta_t ) that quantifies the loss increase caused by a too‑large step size. By embedding a broad family of model‑based updates into a unified proximal formulation, they show that the expected value of ( \delta_t ) directly determines the scaling of the classic sub‑optimality bounds for both average‑iterate and last‑iterate guarantees.

For convex problems, the authors prove two key results. Lemma 2 decomposes the cumulative expected sub‑optimality into a bias term (proportional to the initial distance) and a variance‑like term that involves ( \sum_{t}\alpha_t\mathbb{E}