Statistical-Computational Trade-offs in Learning Multi-Index Models via Harmonic Analysis

We study the problem of learning multi-index models (MIMs), where the label depends on the input $\boldsymbol{x} \in \mathbb{R}^d$ only through an unknown $\mathsf{s}$-dimensional projection $\boldsymbol{W}_*^\mathsf{T} \boldsymbol{x} \in \mathbb{R}^\mathsf{s}$. Exploiting the equivariance of this problem under the orthogonal group $\mathcal{O}_d$, we obtain a sharp harmonic-analytic characterization of the learning complexity for MIMs with spherically symmetric inputs – which refines and generalizes previous Gaussian-specific analyses. Specifically, we derive statistical and computational complexity lower bounds within the Statistical Query (SQ) and Low-Degree Polynomial (LDP) frameworks. These bounds decompose naturally across spherical harmonic subspaces. Guided by this decomposition, we construct a family of spectral algorithms based on harmonic tensor unfolding that sequentially recover the latent directions and (nearly) achieve these SQ and LDP lower bounds. Depending on the choice of harmonic degree sequence, these estimators can realize a broad range of trade-offs between sample and runtime complexity. From a technical standpoint, our results build on the semisimple decomposition of the $\mathcal{O}_d$-action on $L^2 (\mathbb{S}^{d-1})$ and the intertwining isomorphism between spherical harmonics and traceless symmetric tensors.

💡 Research Summary

The paper tackles the fundamental problem of learning multi‑index models (MIMs) in high dimensions when the input distribution is spherically symmetric. A MIM assumes that the response y depends on the covariate x∈ℝᵈ only through an unknown low‑dimensional projection W_*ᵀx ∈ℝˢ, where s ≪ d. While previous works have largely focused on Gaussian inputs and relied on Hermite expansions, this work adopts a representation‑theoretic viewpoint: the orthogonal group 𝒪₍d₎ acts on the sphere 𝕊^{d‑1}, and the space L²(𝕊^{d‑1}) decomposes into irreducible subspaces given by spherical harmonics of degree ℓ. This decomposition is the key to a sharp harmonic‑analytic characterization of learning complexity.

Main theoretical contributions

1. Lower bounds in two complementary frameworks
   Low‑Degree Polynomial (LDP) framework: By expanding the link function ν_d(·|t) in the spherical‑harmonic basis, the authors identify the smallest degree ℓ* with a non‑zero coefficient (the “generative degree”). They prove that any polynomial‑time algorithm that succeeds must use at least Θ(d^{max(1,ℓ*/2)}) samples. This generalizes the Hermite‑based results for Gaussian MIMs to any spherically invariant input.
   Statistical Query (SQ) framework: Here the focus is on query (runtime) complexity. The authors define a “Leap complexity” as the cost of the hardest stage in an optimal multi‑step recovery process. They derive SQ lower bounds that depend on the same harmonic degrees, showing that the query complexity can be substantially larger than the sample complexity when ℓ* is high. The two bounds therefore capture distinct computational barriers.

2. Harmonic tensor unfolding algorithms
   Guided by the harmonic decomposition, the paper introduces a family of spectral algorithms called harmonic tensor unfolding. The data are first projected onto the spherical‑harmonic subspaces, yielding traceless symmetric tensors that are isomorphic to the harmonic components. For a chosen degree ℓ, the algorithm “unfolds’’ the corresponding tensor, performs an eigen‑decomposition, and extracts a latent direction. After removing the contribution of the recovered direction from the data, the procedure repeats for the next degree. By selecting a sequence of degrees (e.g., increasing gradually), the algorithm can trade off sample efficiency against runtime: low‑degree steps are cheap but may require many samples, while high‑degree steps are sample‑efficient but computationally intensive. The authors prove that these procedures achieve (up to polylogarithmic factors) the lower bounds derived in both the LDP and SQ settings, thereby establishing near‑optimality.

3. Applications and extensions
   The framework subsumes several important special cases:

   • Gaussian MIMs: The spherical‑harmonic expansion coincides with the Hermite expansion, recovering known results.
   • Directional MIMs: When the input is uniformly distributed on the sphere, the same analysis applies without modification.
   • Learning polynomials on the sphere: The authors show how their algorithms can be used to estimate arbitrary spherical polynomials, illustrating the breadth of the approach.

Technical underpinnings
The analysis leverages the semisimple decomposition of the 𝒪₍d₎‑action on L²(𝕊^{d‑1}), the intertwining isomorphism between spherical harmonics and traceless symmetric tensors, and concentration tools such as hypercontractivity and matrix concentration inequalities. Detailed proofs are provided for the lower bounds (via moment‑matching arguments and spectral norm calculations) and for the correctness and runtime of the unfolding algorithms (including handling of asymmetric versus symmetric tensor cases).

Discussion
A notable insight is that the SQ and LDP frameworks naturally give rise to two distinct “leap’’ complexities—one governing query cost, the other governing sample cost. The paper argues that no single polynomial‑time algorithm can simultaneously achieve both optimal sample and optimal runtime complexities for general MIMs, highlighting an intrinsic statistical‑computational trade‑off. Moreover, the authors point out that the harmonic‑analysis viewpoint extends beyond the orthogonal group; analogous decompositions exist for other compact groups, suggesting a broader applicability to equivariant learning problems.

Conclusion
By marrying representation theory with modern statistical‑computational lower‑bound techniques, the authors provide a unified, group‑theoretic description of the difficulty of learning multi‑index models under spherical symmetry. Their harmonic tensor unfolding algorithms not only match the derived lower bounds but also offer a flexible toolbox for navigating the sample‑time trade‑off. This work thus bridges the gap between Gaussian‑specific analyses and a fully general spherical setting, opening avenues for further exploration of equivariant learning under other symmetry groups.