High-dimensional covariance estimation based on Gaussian graphical models

Undirected graphs are often used to describe high dimensional distributions. Under sparsity conditions, the graph can be estimated using $\ell_1$-penalization methods. We propose and study the following method. We combine a multiple regression approach with ideas of thresholding and refitting: first we infer a sparse undirected graphical model structure via thresholding of each among many $\ell_1$-norm penalized regression functions; we then estimate the covariance matrix and its inverse using the maximum likelihood estimator. We show that under suitable conditions, this approach yields consistent estimation in terms of graphical structure and fast convergence rates with respect to the operator and Frobenius norm for the covariance matrix and its inverse. We also derive an explicit bound for the Kullback Leibler divergence.

💡 Research Summary

The paper introduces a novel two‑stage procedure, called Gelato (Graph estimation with Lasso and Thresholding), for estimating both the covariance matrix Σ and its inverse (the precision matrix Θ) of a high‑dimensional multivariate Gaussian distribution. In the first stage, the authors perform node‑wise Lasso regressions: for each variable X_i they regress it on all other variables X_{−i} with an ℓ₁ penalty λ, obtaining a sparse coefficient vector β̂_i. Because the Lasso tends to retain many small but non‑zero coefficients, a hard‑threshold τ is applied to set coefficients with absolute value below τ to zero. An edge (i,j) is declared present if either β̂_{ij} or β̂_{ji} survives the threshold (“OR” rule). This yields an estimated undirected graph Ê that is typically much sparser than the one obtained by plain Lasso, reducing false positive edge selections.

In the second stage, the estimated graph is used as a structural constraint in a maximum‑likelihood estimation of the precision matrix. The sample covariance S_n is standardized to a correlation matrix Γ_n, and the constrained MLE solves
 min_{Θ ≻ 0, Θ_{ij}=0 for (i,j)∉Ê} tr(ΘΓ_n) – log|Θ|.
If the graph is sufficiently sparse, the constrained MLE exists and is unique without additional penalization. The authors provide a thorough theoretical analysis under a set of assumptions: a restricted eigenvalue condition on the design, sub‑Gaussian tails, and a notion of “essential sparsity” S₀,n that counts only those off‑diagonal entries of Θ₀ that are larger than a threshold proportional to √(log p / n). Their main results are:

1. Graph selection consistency – with appropriate λ and τ, the estimated edge set Ê equals the true edge set E₀ with probability tending to one.
2. Fast convergence rates – both the precision and covariance estimators achieve Frobenius‑norm error O(√(S₀,n log p / n)) and similar operator‑norm bounds, which are strictly better than the rates typically proved for the graphical Lasso (GLasso) that depend on the total number of edges.
3. Kullback‑Leibler risk bound – an explicit upper bound on the KL divergence between the true and estimated Gaussian distributions is derived, establishing risk consistency.
4. Bias‑variance trade‑off – the paper shows that the bias introduced by thresholding is controlled by S₀,n, while the variance is reduced thanks to the sparsity of the graph.

The authors also discuss practical aspects: λ and τ are chosen via cross‑validation (λ by minimizing prediction error of the node‑wise regressions, τ by maximizing the multivariate Gaussian log‑likelihood on a validation set). Computationally, Gelato requires solving p Lasso problems and a single constrained MLE, yielding a runtime comparable to GLasso and far simpler than methods that involve iterative penalized likelihood updates.

Empirical experiments on synthetic data and real‑world datasets (e.g., gene expression networks) demonstrate that Gelato often recovers the true graph more accurately than GLasso, Adaptive GLasso, and the SPACE algorithm, especially when the true precision matrix contains many small but non‑zero entries that would be missed or falsely retained by other methods. Moreover, the covariance and precision estimates exhibit lower Frobenius and operator norm errors, confirming the theoretical advantages.

In summary, Gelato provides a conceptually simple yet theoretically robust framework for high‑dimensional covariance estimation: a sparsity‑inducing Lasso‑thresholding step for reliable graph selection followed by a constrained maximum‑likelihood step that yields optimal statistical rates under weaker assumptions than many existing approaches. This makes it a compelling alternative for practitioners dealing with large‑scale Gaussian graphical models.