Autotune: fast, accurate, and automatic tuning parameter selection for Lasso

Least absolute shrinkage and selection operator (Lasso), a popular method for high-dimensional regression, is now used widely for estimating high-dimensional time series models such as the vector autoregression (VAR). Selecting its tuning parameter efficiently and accurately remains a challenge, despite the abundance of available methods for doing so. We propose $\mathsf{autotune}$, a strategy for Lasso to automatically tune itself by optimizing a penalized Gaussian log-likelihood alternately over regression coefficients and noise standard deviation. Using extensive simulation experiments on regression and VAR models, we show that $\mathsf{autotune}$ is faster, and provides better generalization and model selection than established alternatives in low signal-to-noise regimes. In the process, $\mathsf{autotune}$ provides a new estimator of noise standard deviation that can be used for high-dimensional inference, and a new visual diagnostic procedure for checking the sparsity assumption on regression coefficients. Finally, we demonstrate the utility of $\mathsf{autotune}$ on a real-world financial data set. An R package based on C++ is also made publicly available on Github.

💡 Research Summary

The Lasso (Least Absolute Shrinkage and Selection Operator) has become a cornerstone of high-dimensional regression analysis, particularly in the context of complex models like Vector Autoregression (VAR). A critical component of Lasso’s effectiveness is the selection of the tuning parameter ($\lambda$), which controls the degree of shrinkage and variable selection. Despite numerous existing methods, finding an optimal $\lambda$ that balances computational efficiency with predictive accuracy remains a significant challenge, especially in high-dimensional settings where the signal-to-noise ratio (SNR) is low.

This paper introduces autotune, a novel strategy designed to automate the tuning process of the Lasso. The core innovation of autotune lies in its approach to optimizing a penalized Gaussian log-likelihood. Unlike traditional methods that focus solely on $\lambda$, autotune employs an alternating optimization scheme that iteratively updates both the regression coefficients ($\beta$) and the noise standard deviation ($\sigma$). By treating the noise level as a parameter to be optimized alongside the coefficients, the method achieves a more integrated and statistically sound tuning process.

Extensive simulation experiments conducted on both standard regression and VAR models demonstrate the superiority of autotune. The results indicate that autotune is not only computationally faster than established alternatives but also provides significantly better model selection and generalization performance. Notably, its advantages are most pronounced in low SNR regimes, where distinguishing signal from noise is notoriously difficult. This makes autotune particularly robust for analyzing datasets characterized by high levels of uncertainty.

Beyond parameter tuning, the paper presents two significant secondary contributions. First, the autotune framework yields a new estimator for the noise standard deviation, which holds great potential for high-dimensional statistical inference. Second, the authors propose a new visual diagnostic procedure to assess the validity of the sparsity assumption on regression coefficients, allowing researchers to visually inspect whether the model’s structural assumptions hold true.

The practical utility of autotune is further validated through its application to a real-world financial dataset, demonstrating its effectiveness in handling complex, noisy, and high-dimensional time-series data. To facilitate widespread adoption in the scientific community, the authors have released an R package, implemented in C++ for high performance, via GitHub. This tool provides a powerful, ready-to-use solution for practitioners dealing with the complexities of high-dimensional regression and time-series modeling.