Dynamic Momentum Recalibration in Online Gradient Learning

Stochastic Gradient Descent (SGD) and its momentum variants form the backbone of deep learning optimization, yet the underlying dynamics of their gradient behavior remain insufficiently understood. In this work, we reinterpret gradient updates through the lens of signal processing and reveal that fixed momentum coefficients inherently distort the balance between bias and variance, leading to skewed or suboptimal parameter updates. To address this, we propose SGDF (SGD with Filter), an optimizer inspired by the principles of Optimal Linear Filtering. SGDF computes an online, time-varying gain to dynamically refine gradient estimation by minimizing the mean-squared error, thereby achieving an optimal trade-off between noise suppression and signal preservation. Furthermore, our approach could extend to other optimizers, showcasing its broad applicability to optimization frameworks. Extensive experiments across diverse architectures and benchmarks demonstrate SGDF surpasses conventional momentum methods and achieves performance on par with or surpassing state-of-the-art optimizers.

💡 Research Summary

This paper, titled “Dynamic Momentum Recalibration in Online Gradient Learning,” presents a critical analysis of fundamental limitations in momentum-based stochastic optimization and introduces a novel optimizer, SGDF (SGD with Filter), to address them.

The authors begin by identifying a core dilemma in deep learning optimization: the inherent bias-variance trade-off in gradient estimation used by optimizers like SGD with momentum. They argue that fixed momentum coefficients, as used in Exponential Moving Average (EMA) and Classical Momentum (CM), are inherently suboptimal. Through a unified stochastic differential equation (SDE) framework, they theoretically demonstrate that these static methods suffer from “parameter-shift bias,” which becomes unbounded as the momentum factor (β) approaches 1, leading to skewed or suboptimal parameter updates (Theorem 2.3, Table 1).

To overcome this rigid trade-off, the paper proposes SGDF, an optimizer inspired by Optimal Linear Filtering principles from signal processing. The core innovation of SGDF is its online, time-varying gain (K_t). Instead of using a fixed rule to blend past and present gradients, SGDF dynamically computes the optimal blending weight by minimizing the mean-squared error of the gradient estimate. The algorithm maintains bias-corrected first-moment (m_t) and a second-moment estimate (s_t) representing the variance of the momentum estimator. The key gain is calculated as K_t = Estimated_Variance(m_t) /