It's all In the (Exponential) Family: An Equivalence between Maximum Likelihood Estimation and Control Variates for Sketching Algorithms

Maximum likelihood estimators (MLE) and control variate estimators (CVE) have been used in conjunction with known information across sketching algorithms and applications in machine learning. We prove that under certain conditions in an exponential family, an optimal CVE will achieve the same asymptotic variance as the MLE, giving an Expectation-Maximization (EM) algorithm for the MLE. Experiments show the EM algorithm is faster and numerically stable compared to other root finding algorithms for the MLE for the bivariate Normal distribution, and we expect this to hold across distributions satisfying these conditions. We show how the EM algorithm leads to reproducibility for algorithms using MLE / CVE, and demonstrate how the EM algorithm leads to finding the MLE when the CV weights are known.

💡 Research Summary

The paper investigates the relationship between maximum likelihood estimation (MLE) and control variate estimation (CVE) within the framework of exponential family distributions, a setting that underlies many sketching algorithms and machine‑learning applications. After reviewing the canonical exponential‑family representation (p(X|\nu)=\exp{\eta(\nu)^{\top}y(X)-\psi(\eta(\nu))}g(X)), the authors separate the unknown parameters (\nu_E) from the known parameters (\nu_K). They assume that each mean parameter (\mu_i=\partial\psi/\partial\eta_i) coincides with the corresponding natural parameter (\nu_i).

Using the well‑known identities (d\mu = V,d\eta) and (d\eta = V^{-1} d\mu) (where (V) is the covariance matrix of the sufficient statistics), they derive the Fisher information for (\nu_E) and the asymptotic variance of the MLE. In parallel, they formulate a CVE of the form (Z = X + \sum_{i=1}^k c_i(Y_i - \mathbb{E}