Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning

Contrastive Language-Image Pre-training (CLIP) has become a cornerstone in vision-language representation learning, powering diverse downstream tasks and serving as the default vision backbone in multimodal large language models (MLLMs). Despite its success, CLIP’s dense and opaque latent representations pose significant interpretability challenges. A common assumption is that interpretability and performance are in tension: enforcing sparsity during training degrades accuracy, motivating recent post-hoc approaches such as Sparse Autoencoders (SAEs). However, these post-hoc approaches often suffer from degraded downstream performance and loss of CLIP’s inherent multimodal capabilities, with most learned features remaining unimodal. We propose a simple yet effective approach that integrates sparsity directly into CLIP training, yielding representations that are both interpretable and performant. Compared to SAEs, our Sparse CLIP representations preserve strong downstream task performance, achieve superior interpretability, and retain multimodal capabilities. We show that multimodal sparse features enable straightforward semantic concept alignment and reveal training dynamics of how cross-modal knowledge emerges. Finally, as a proof of concept, we train a vision-language model on sparse CLIP representations that enables interpretable, vision-based steering capabilities. Our findings challenge conventional wisdom that interpretability requires sacrificing accuracy and demonstrate that interpretability and performance can be co-optimized, offering a promising design principle for future models.

💡 Research Summary

The paper tackles a fundamental tension in modern vision‑language models: CLIP’s dense, high‑dimensional embeddings are powerful but opaque, while post‑hoc sparsification methods such as Sparse Autoencoders (SAEs) improve interpretability at the cost of downstream performance and multimodal fidelity. The authors propose “Sparse CLIP,” a simple modification to the original CLIP training pipeline that injects sparsity directly during contrastive learning, thereby achieving both interpretability and state‑of‑the‑art performance.
Two key changes are introduced: (1) a non‑negative constraint by applying a ReLU after the final projection layer, and (2) a dramatic expansion of the projection dimensionality (e.g., from a few hundred to >55 k dimensions for a ViT‑L/14 model). Theoretical work linking non‑negative contrastive learning to non‑negative matrix factorization (NMF) justifies why these changes induce a sparse, dictionary‑like representation. Small‑scale ablations on ViT‑B/32 with 15 M image‑text pairs reveal that sparsity and accuracy improve only when both components are present. ReLU yields a gradual, learning‑friendly sparsity formation, whereas L1 regularization or hard Top‑K gating suppresses activations too early and harms performance.
A further control knob is the logit‑scale (temperature) cap; lowering this cap reduces L0 sparsity but, if set too low, degrades zero‑shot accuracy. The authors thus identify a sweet‑spot range that balances sparsity and capacity.
Scaling up, they train a ViT‑L/14 model on the full 2.2 B MetaCLIP dataset for six epochs, using a 72× expansion factor (55 296 dimensions). Two variants—Sparse (logit‑scale cap = 50, L0 ≈ 0.66 %) and Sparse+ (cap = 40, L0 ≈ 0.47 %)—are evaluated on a battery of zero‑shot benchmarks (ImageNet‑1k, ImageNet‑v2, ObjectNet, ImageNet‑A/R/S, fine‑grained datasets). Both models match or slightly exceed the dense baseline on classification tasks (average gains of +0.5 % to +0.7 % absolute), and outperform on bounding‑box classification, while showing a modest drop on COCO image‑text retrieval, likely because the sparse representations focus on the dominant visual subject.
Interpretability analyses demonstrate that individual sparse dimensions align cleanly with semantic concepts across modalities; probing with textual prompts yields high cosine similarity for dimensions representing “animals,” “food,” “landscape,” etc. The authors visualize the emergence of these multimodal concepts during training, revealing how cross‑modal knowledge propagates from early to later layers.
Finally, they build a vision‑language model that consumes Sparse CLIP embeddings as its visual backbone. This model enables an interpretable, vision‑based steering interface: a user can issue textual commands that map to specific sparse dimensions, which in turn drive robot actions. The demonstration underscores the practical utility of having a compact, human‑readable visual codebook.
Overall, the work disproves the prevailing belief that sparsity inevitably harms performance. By integrating non‑negativity and high‑dimensional projection into the contrastive objective, Sparse CLIP delivers a representation that is simultaneously sparse, multimodal, and high‑performing, offering a new design principle for future interpretable multimodal models.