Provable Target Sample Complexity Improvements as Pre-Trained Models Scale

Pre-trained models have become indispensable for efficiently building models across a broad spectrum of downstream tasks. The advantages of pre-trained models have been highlighted by empirical studies on scaling laws, which demonstrate that larger pre-trained models can significantly reduce the sample complexity of downstream learning. However, existing theoretical investigations of pre-trained models lack the capability to explain this phenomenon. In this paper, we provide a theoretical investigation by introducing a novel framework, caulking, inspired by parameter-efficient fine-tuning (PEFT) methods such as adapter-based fine-tuning, low-rank adaptation, and partial fine-tuning. Our analysis establishes that improved pre-trained models provably decrease the sample complexity of downstream tasks, thereby offering theoretical justification for the empirically observed scaling laws relating pre-trained model size to downstream performance, a relationship not covered by existing results.

💡 Research Summary

**
The paper tackles a fundamental question raised by recent empirical scaling laws: why do larger pre‑trained models dramatically reduce the amount of downstream data needed to achieve a given performance? Existing theoretical analyses typically bound the downstream error by a sum of two terms, one decreasing with the source sample size (m) (the amount of data used to pre‑train) and one decreasing with the target sample size (n). In those bounds the exponent governing the dependence on (n) (often denoted (\beta)) is independent of (m), so they cannot explain the observed improvement in target‑sample complexity as the pre‑trained model grows.

To fill this gap the authors introduce a new conceptual framework called caulking, inspired by parameter‑efficient fine‑tuning (PEFT) techniques such as adapters, LoRA, and partial fine‑tuning. A pre‑trained model is decomposed into a feature extractor (g_e) and a head (g_h). The target task’s optimal predictor (f^{*}) is assumed to be caulkable, meaning that there exists an intermediate function (g_a) (the adapter) such that
\