Cross-Modal Redundancy and the Geometry of Vision-Language Embeddings

Vision-language models (VLMs) align images and text with remarkable success, yet the geometry of their shared embedding space remains poorly understood. To probe this geometry, we begin from the Iso-Energy Assumption, which exploits cross-modal redundancy: a concept that is truly shared should exhibit the same average energy across modalities. We operationalize this assumption with an Aligned Sparse Autoencoder (SAE) that encourages energy consistency during training while preserving reconstruction. We find that this inductive bias changes the SAE solution without harming reconstruction, giving us a representation that serves as a tool for geometric analysis. Sanity checks on controlled data with known ground truth confirm that alignment improves when Iso-Energy holds and remains neutral when it does not. Applied to foundational VLMs, our framework reveals a clear structure with practical consequences: (i) sparse bimodal atoms carry the entire cross-modal alignment signal; (ii) unimodal atoms act as modality-specific biases and fully explain the modality gap; (iii) removing unimodal atoms collapses the gap without harming performance; (iv) restricting vector arithmetic to the bimodal subspace yields in-distribution edits and improved retrieval. These findings suggest that the right inductive bias can both preserve model fidelity and render the latent geometry interpretable and actionable.

💡 Research Summary

The paper tackles the largely unexplored geometry of shared embedding spaces in vision‑language models (VLMs). While dual‑encoder models such as CLIP and SigLIP successfully align images and text, the internal organization that yields this alignment—and the persistent “modality gap” between visual and textual embeddings—remains opaque. The authors introduce two central ideas to illuminate this structure.

First, the Iso‑Energy Assumption posits that a truly shared semantic concept should exhibit the same average squared activation (energy) in both modalities. Formally, for each latent atom k the second‑moment of its code value must be invariant across domains (image vs. text). This statistical symmetry provides a minimal cross‑modal constraint that makes the otherwise ill‑posed nonlinear ICA problem of recovering latent concepts more identifiable.

Second, they operationalize the assumption with an Aligned Sparse Autoencoder (Aligned SAE). Building on a Matching‑Pursuit‑based sparse autoencoder, they add a soft alignment loss
(L_{\text{align}} = -\frac{1}{b}\operatorname{Tr}(Z^{(I)} Z^{(T)^\top}))
to the usual reconstruction‑plus‑sparsity objective. The total loss is (L = L_{\text{SAE}} + \beta L_{\text{align}}) with a tiny weight (\beta \approx 10^{-4}). This term encourages the activations of the same atom to have similar magnitude across a batch of image‑text pairs, without degrading reconstruction quality.

The experimental program proceeds in two stages. In a synthetic setting, the authors construct datasets where ground‑truth concepts are either truly bimodal (present in both modalities) or modality‑specific. The Aligned SAE improves cross‑modal alignment scores only when the Iso‑Energy condition holds, confirming that the regularizer selectively promotes shared atoms while remaining neutral otherwise.

In the main study, the method is applied to large‑scale pretrained VLMs (e.g., CLIP‑ViT‑B/32, SigLIP‑ResNet‑50). After training the Aligned SAE on the frozen encoder outputs, several striking phenomena emerge:

1. Sparse Bimodal Atoms Carry All Alignment Signal – Only a small subset of atoms (tens) are active in both image and text; these atoms account for virtually the entire contrastive alignment performance.

2. Unimodal Atoms Explain the Modality Gap – The majority of high‑energy atoms fire exclusively in one modality. Their collective contribution reproduces the mean‑difference vector (\Delta = \mu_I - \mu_T) that underlies the well‑known cone‑shaped modality gap.

3. Removing Unimodal Atoms Collapses the Gap – Zero‑ing or filtering out modality‑specific atoms forces image and text embeddings into the same subspace, dramatically reducing (|\Delta|) (by >90 %) while preserving or even slightly improving retrieval accuracy.

4. Vector Arithmetic Restricted to the Bimodal Subspace Yields In‑Distribution Edits – Classic word‑vector style operations (e.g., “dog − animal + cat”) performed only on bimodal dimensions stay within the data manifold, leading to more realistic image‑text edits and higher downstream performance compared to using the full embedding.

These results refine prior explanations of the modality gap that relied solely on geometric cone effects or mean‑shifts. The authors explicitly decompose the embedding space into three linear subspaces: a shared subspace (\Gamma) spanned by bimodal atoms, and modality‑specific subspaces (\Omega_I) and (\Omega_T) spanned by unimodal atoms. This decomposition is both descriptive (it matches observed statistics) and prescriptive (it enables targeted interventions).

Importantly, the Aligned SAE retains the reconstruction fidelity of a standard SAE, proving that the Iso‑Energy bias does not compromise the autoencoder’s primary task. Instead, it provides a principled, quantitative tool for diagnosing and manipulating cross‑modal structure. The paper demonstrates that a simple energy‑consistency regularizer can turn an otherwise opaque high‑dimensional embedding into an interpretable, actionable representation.

Future directions suggested include extending the Iso‑Energy regularizer to other multimodal pairings (video‑text, audio‑text), integrating the alignment term earlier in the training pipeline, or dynamically adapting (\beta) to balance reconstruction and alignment throughout training. Overall, the work offers a compelling blend of theory, methodology, and empirical validation that advances our understanding of how vision‑language models internally align concepts across modalities.