Deep Learning of Compositional Targets with Hierarchical Spectral Methods

Why depth yields a genuine computational advantage over shallow methods remains a central open question in learning theory. We study this question in a controlled high-dimensional Gaussian setting, focusing on compositional target functions. We analyze their learnability using an explicit three-layer fitting model trained via layer-wise spectral estimators. Although the target is globally a high-degree polynomial, its compositional structure allows learning to proceed in stages: an intermediate representation reveals structure that is inaccessible at the input level. This reduces learning to simpler spectral estimation problems, well studied in the context of multi-index models, whereas any shallow estimator must resolve all components simultaneously. Our analysis relies on Gaussian universality, leading to sharp separations in sample complexity between two and three-layer learning strategies.

💡 Research Summary

The paper investigates why depth can provide a genuine computational advantage over shallow methods by studying a controlled high‑dimensional Gaussian setting with compositional target functions. The authors consider target functions that are globally high‑degree polynomials but are built as a composition of two nonlinear polynomial blocks. Specifically, the input $x\in\mathbb{R}^d$ is first transformed into a $d^{\varepsilon}$‑dimensional intermediate representation $h^{(1)}$ via inner products with random Gaussian tensors $A^{(1)}_i$ and degree‑$k$ Hermite polynomials $H_k(x)$. A second random tensor $A^{(2)}$ together with degree‑2 Hermite polynomials $H_2(\cdot)$ maps $h^{(1)}$ to a scalar $h^{(2)}$, and a final non‑linearity $g^{\star}$ (a polynomial of degree $p$) produces the label $y=g^{\star}(h^{(2)})$. Although $f^{\star}(x)$ is a high‑order polynomial in $x$, its compositional structure allows learning to be broken into two simpler stages.

Instead of analyzing gradient descent, the authors propose an explicit layer‑wise spectral learning procedure. In the first stage they compute flattened degree‑$k$ Hermite features for each sample, form the empirical covariance (moment) matrix, and recover the $d^{\varepsilon}$ signal directions by a PCA‑type eigen‑decomposition. Using results from random matrix theory (Baik–Ben Arous–Péché transition), they show that when the number of samples satisfies $n\gg d^{k+\varepsilon}$ and the signal subspace is sufficiently sparse ($d^{\varepsilon}\ll d^{k}$), the top eigenvectors separate from the bulk and can be estimated consistently. This yields an accurate estimate of $h^{(1)}$.

Conditioned on the recovered $h^{(1)}$, the second stage repeats the same spectral method on degree‑2 Hermite features of $h^{(1)}$, requiring only $n\gg d^{2\varepsilon}$ samples to estimate the scalar $h^{(2)}$. The final one‑dimensional nonlinearity $g^{\star}$ can then be fitted with negligible additional data because the effective dimension has collapsed to one.

Consequently, the total sample complexity of the three‑layer hierarchical procedure is dominated by the first stage and scales as $O(d^{k+\varepsilon})$. For the concrete case $k=2$, this becomes $O(d^{2+\varepsilon})$, independent of the outer polynomial degree $p$. By contrast, kernel methods or random‑feature models can only learn low‑degree polynomial approximations of $f^{\star}$ and require $n=O(d^{4p})$ samples, reflecting a dramatic separation between shallow and deep strategies.

A further contribution is a new Gaussian Equivalence Principle for hierarchical models: at each layer, after appropriate normalization, the learned representations behave asymptotically like Gaussian vectors with explicitly computable covariances. This extends previous universality results from shallow to multi‑layer settings and justifies the spectral analysis even when the underlying tensors are non‑Gaussian.

Theoretical results are proved rigorously for $\varepsilon<1/2$, relying on bounded operator norms of the estimators. Extensive simulations confirm the predicted phase transitions and sample‑complexity scalings, even in moderate dimensions and beyond the proven regime. The experiments illustrate that the hierarchical spectral algorithm successfully recovers the intermediate features and achieves accurate prediction with far fewer samples than shallow baselines.

In summary, the paper provides a clear, mathematically grounded explanation of how depth enables progressive disentanglement of compositional structure, turning a globally complex learning problem into a sequence of low‑order spectral estimation tasks. This yields optimal (up to constants) sample complexity for a broad class of hierarchical polynomial targets and highlights the power of depth beyond mere expressive capacity, offering a transparent alternative to gradient‑based analyses.