Low-Rank Plus Sparse Matrix Transfer Learning under Growing Representations and Ambient Dimensions

Learning systems often expand their ambient features or latent representations over time, embedding earlier representations into larger spaces with limited new latent structure. We study transfer learning for structured matrix estimation under simultaneous growth of the ambient dimension and the intrinsic representation, where a well-estimated source task is embedded as a subspace of a higher-dimensional target task. We propose a general transfer framework in which the target parameter decomposes into an embedded source component, low-dimensional low-rank innovations, and sparse edits, and develop an anchored alternating projection estimator that preserves transferred subspaces while estimating only low-dimensional innovations and sparse modifications. We establish deterministic error bounds that separate target noise, representation growth, and source estimation error, yielding strictly improved rates when rank and sparsity increments are small. We demonstrate the generality of the framework by applying it to two canonical problems. For Markov transition matrix estimation from a single trajectory, we derive end-to-end theoretical guarantees under dependent noise. For structured covariance estimation under enlarged dimensions, we provide complementary theoretical analysis in the appendix and empirically validate consistent transfer gains.

💡 Research Summary

The paper addresses a realistic scenario in modern machine‑learning systems where the feature space or latent representation expands over time—new sensors, modalities, or model components are added while preserving earlier features as identifiable subspaces of the enlarged space. In this “representation growth” setting, a well‑estimated source task lives in a low‑dimensional ambient space, and the target task occupies a strictly larger space that embeds the source subspace via a zero‑padding operator. The target parameter is modeled as the sum of three components: (i) the embedded source low‑rank matrix, (ii) a low‑dimensional low‑rank innovation orthogonal to the embedded subspace, and (iii) a sparse edit that modifies only a few entries relative to the source sparse component. This decomposition captures the intuition that expanding the representation typically introduces only a few new latent directions and a limited number of localized changes.

To exploit this structure, the authors propose an “anchored alternating projection” estimator. The algorithm alternates between (a) an anchored low‑rank projection that preserves the embedded source row and column subspaces while estimating the innovation subspace (of rank δ_r,2), and (b) a sparse‑edit projection that retains at most δ_s,2 non‑zero deviations from the transferred sparse anchor. The low‑rank projection is implemented by first projecting the current residual onto the orthogonal complement of the source subspaces, extracting the leading δ_r,2 singular vectors, and then forming the updated low‑rank factors that explicitly contain the source bases. The sparse‑edit projection simply adds the top‑δ_s,2 magnitude entries of the residual to the transferred sparse matrix. Repeating these two steps yields a solution to the constrained optimization problem that minimizes the Frobenius norm of the reconstruction error while respecting the anchored structure.

The theoretical contribution is a deterministic error bound that cleanly separates three sources of error: (1) intrinsic target noise, (2) approximation error due to representation growth (i.e., the mismatch between the true target subspaces and the embedded source subspaces), and (3) source estimation error (since the source matrix is estimated from abundant data, not known exactly). By carefully aligning the true low‑rank factors with the estimated ones to eliminate rotational ambiguity, the authors show that the overall error scales as \