Function-Space Empirical Bayes Regularisation with Large Vision-Language Model Priors

Bayesian deep learning (BDL) provides a principled framework for reliable uncertainty quantification by combining deep neural networks with Bayesian inference. A central challenge in BDL lies in the design of informative prior distributions that scale effectively to high-dimensional data. Recent functional variational inference (VI) approaches address this issue by imposing priors directly in function space; however, most existing methods rely on Gaussian process (GP) priors, whose expressiveness and generalisation capabilities become limited in high-dimensional regimes. In this work, we propose VLM-FS-EB, a novel function-space empirical Bayes regularisation framework, leveraging large vision-language models (VLMs) to generates semantically meaningful context points. These synthetic samples are then used VLMs for embeddings to construct expressive functional priors. Furthermore, the proposed method is evaluated against various baselines, and experimental results demonstrate that our method consistently improves predictive performance and yields more reliable uncertainty estimates, particularly in out-of-distribution (OOD) detection tasks and data-scarce regimes.

💡 Research Summary

This paper introduces VLM-FS-EB, a novel framework that integrates large Vision-Language Models (VLMs) into the Function-Space Empirical Bayes (FS-EB) paradigm to address core challenges in Bayesian Deep Learning (BDL). The central problem in BDL is designing informative prior distributions that scale effectively to high-dimensional data. While function-space variational inference (FSVI) methods place priors directly over functions, they predominantly rely on Gaussian Process (GP) priors, whose expressiveness and generalization capabilities are limited in high-dimensional regimes.

The proposed VLM-FS-EB framework leverages the dual capabilities of modern VLMs to overcome these limitations. First, it utilizes generative VLMs (e.g., encoder-decoder models) to synthesize diverse, semantically meaningful context points in a controllable, data-free manner. This eliminates the framework’s dependency on external context data, which is a critical bottleneck in data-scarce scenarios. Second, it employs frozen contrastive VLMs (e.g., CLIP) as powerful feature extractors. The embeddings from these models, pre-trained on billions of image-text pairs, encapsulate rich semantic structures and relationships, providing a much more expressive basis for constructing the functional prior than standard handcrafted kernels or learned features from scratch.

Methodologically, VLM-FS-EB builds upon the FS-EB approach. FS-EB defines an empirical prior by solving an auxiliary inference problem on a set of context points x_c using a stochastic linear model defined over a feature extractor h(·). VLM-FS-EB innovates by using a generative VLM to produce x_c and a contrastive VLM to implement h(·). This results in a semantically informed prior that regularizes the model in both parameter and function space without requiring linearization approximations, thus avoiding associated errors and instability.

The paper presents comprehensive experiments on multiple real-world image classification benchmarks (e.g., CIFAR-10, CIFAR-100). The results demonstrate that VLM-FS-EB consistently outperforms a range of strong baselines, including parameter-space regularization methods and prior FSVI techniques. The improvements are particularly pronounced in two challenging settings: 1) Data-scarce regimes (e.g., few-shot learning), where the ability to generate relevant context and use powerful pre-trained features is most valuable, and 2) Out-of-Distribution (OOD) detection, where the semantically structured prior leads to better-calibrated and more reliable uncertainty estimates. The work concludes that leveraging the knowledge embedded within large foundation models through a principled Bayesian function-space framework is a highly effective strategy for building more data-efficient and trustworthy deep learning systems.