Statistical Learning for Latent Embedding Alignment with Application to Brain Encoding and Decoding

Brain encoding and decoding aims to understand the relationship between external stimuli and brain activities, and is a fundamental problem in neuroscience. In this article, we study latent embedding alignment for brain encoding and decoding, with a focus on improving sample efficiency under limited fMRI-stimulus paired data and substantial subject heterogeneity. We propose a lightweight alignment framework equipped with two statistical learning components: inverse semi-supervised learning that leverages abundant unpaired stimulus embeddings through inverse mapping and residual debiasing, and meta transfer learning that borrows strength from pretrained models across subjects via sparse aggregation and residual correction. Both methods operate exclusively at the alignment stage while keeping encoders and decoders frozen, allowing for efficient computation, modular deployment, and rigorous theoretical analysis. We establish finite-sample generalization bounds and safety guarantees, and demonstrate competitive empirical performance on the large-scale fMRI-image reconstruction benchmark data.

💡 Research Summary

The paper tackles the problem of brain encoding and decoding, specifically the visual reconstruction of natural images from fMRI recordings, by focusing on the latent embedding alignment stage. Recognizing that collecting paired fMRI‑image data is costly and that substantial inter‑subject variability hampers generalization, the authors propose a lightweight alignment framework that leaves the image encoder, fMRI encoder, and decoder frozen while improving the mapping between their latent spaces. The framework comprises two novel statistical learning components.

Inverse Semi‑Supervised Learning (ISL) addresses the asymmetry that abundant stimulus embeddings (Y) are available, but paired fMRI embeddings (X) are scarce. In addition to the limited paired set {(X_i, Y_i)}{i=1}^n, a large pool of unpaired stimulus embeddings {Y_j}{j=n+1}^{n+N} is generated by the image encoder. ISL learns an inverse mapping g_φ : Y → X̂ using the unpaired Y’s, then constructs pseudo‑predictors X̂ for the paired Y’s. The residual Δ_i = X_i – X̂_i is used as a bias‑correction term in the loss for the alignment model f_θ. This effectively injects information from the abundant stimulus domain into the scarce predictor domain, yielding a generalization error that scales with 1/√(n+N) rather than 1/√n. The authors provide a finite‑sample risk bound that incorporates the sub‑Gaussian noise assumption and the ℓ_q (q∈