In-Context Semi-Supervised Learning

There has been significant recent interest in understanding the capacity of Transformers for in-context learning (ICL), yet most theory focuses on supervised settings with explicitly labeled pairs. In practice, Transformers often perform well even when labels are sparse or absent, suggesting crucial structure within unlabeled contextual demonstrations. We introduce and study in-context semi-supervised learning (IC-SSL), where a small set of labeled examples is accompanied by many unlabeled points, and show that Transformers can leverage the unlabeled context to learn a robust, context-dependent representation. This representation enables accurate predictions and markedly improves performance in low-label regimes, offering foundational insights into how Transformers exploit unlabeled context for representation learning within the ICL framework.

💡 Research Summary

In this paper the authors introduce a novel learning paradigm called In‑Context Semi‑Supervised Learning (IC‑SSL), which extends the standard in‑context learning (ICL) framework of Transformers to settings where only a few labeled examples are available while a large pool of unlabeled data accompanies them. The key insight is that the “context” in ICL can be broadened to include both labeled and unlabeled tokens, allowing the model to extract geometric structure from the unlabeled portion and use it to build a robust, context‑dependent representation that improves downstream prediction in low‑label regimes.

The proposed architecture consists of two stages that are implemented as sub‑modules of a single Transformer and trained end‑to‑end. The first stage, denoted TF_rep, is responsible for representation learning from the entire set of inputs. It is further split into TF_L, which computes an affinity matrix using an RBF kernel on the raw input coordinates and then forms a discrete Laplacian (\hat L = I - D^{-1}A); and TF_ϕ, which takes (\hat L) and performs power‑iteration‑style updates to obtain the eigenvectors (an Eigenmap) of the Laplacian. This stage therefore reproduces the classic Laplacian Eigenmaps algorithm inside the forward pass of a Transformer, yielding a feature matrix (\Phi = (\phi(x^{(1)}),\dots,\phi(x^{(n)}))) where each (\phi(x^{(i)})) encodes not only the local coordinates of token (i) but also global manifold information derived from all tokens.

The second stage, TF_sup, receives the context‑dependent features together with the few available labels. It learns a linear classifier (f) (followed by a soft‑max) parameterized by class embeddings (w_c). The authors prove that the forward computation of TF_sup is mathematically equivalent to performing a kernelized gradient‑descent update on the classifier parameters, i.e., the attention and MLP layers instantiate an implicit optimization algorithm. Consequently, the model can predict the labels of all unlabeled tokens by effectively “training” on the labeled subset during inference, without any external fine‑tuning.

Training is performed by minimizing a cross‑entropy loss over the unlabeled tokens (for which ground‑truth labels are known during training). The loss depends on the parameters of both stages, the class embeddings, and the representation (\phi), which is itself a function of all inputs. This end‑to‑end objective forces the model to shape (\phi) so that the manifold structure of the data is captured and the subsequent gradient‑descent‑style inference yields accurate predictions.

Empirical evaluation spans a wide range of domains and dimensionalities. Synthetic manifolds in (\mathbb{R}^3) (sphere, cylinder, cone, Swiss‑roll, torus) demonstrate that IC‑SSL dramatically outperforms standard ICL, offline Laplacian Eigenmaps followed by a linear classifier, and other strong baselines, especially when the number of labeled examples is between one and five. Higher‑dimensional product manifolds in (\mathbb{R}^{15}) show the same trend. The method is also tested on a high‑dimensional image manifold generated by Stable Diffusion v1.5, where it achieves >85 % accuracy with only three labeled images, surpassing many‑shot prompting baselines. Finally, on ImageNet‑100 features extracted from a pretrained ResNet‑50, IC‑SSL yields a 12 % absolute gain in top‑1 accuracy over prompt‑based ICL and remains robust when the labeled set is extremely small.

Beyond raw performance, the authors conduct a series of analyses that reveal the inductive bias introduced by the two‑stage design. The first stage imposes a local, sparse‑attention mechanism that naturally computes spectral embeddings; the second stage leverages depth to implement gradient‑based learning, explaining why deeper Transformers are essential for semi‑supervised ICL. Visualization of the learned (\phi) shows that samples of the same class form coherent clusters on the underlying manifold, confirming that the model learns geometry‑aware representations rather than merely memorizing training points.

In summary, the paper makes three major contributions: (1) it defines the IC‑SSL problem and demonstrates that Transformers can exploit unlabeled context for representation learning; (2) it provides a concrete two‑stage Transformer construction that explicitly implements Laplacian Eigenmaps and gradient descent within the forward pass, offering a transparent mechanistic account of how attention and MLP layers realize semi‑supervised ICL; and (3) it validates the approach across synthetic, product‑space, diffusion‑image, and real‑world vision datasets, showing consistent gains in sample efficiency, generalization, and transferability. The work opens new avenues for leveraging abundant unlabeled data in large language and vision models, reducing reliance on curated prompts, and suggests that future research could extend the framework to multi‑label, hierarchical, or sequential output spaces, as well as to massive pretrained models where the same principles may further improve few‑shot performance.