Pool-based Active Learning as Noisy Lossy Compression: Characterizing Label Complexity via Finite Blocklength Analysis

This paper proposes an information-theoretic framework for analyzing the theoretical limits of pool-based active learning (AL), in which a subset of instances is selectively labeled. The proposed framework reformulates pool-based AL as a noisy lossy compression problem by mapping pool observations to noisy symbol observations, data selection to compression, and learning to decoding. This correspondence enables a unified information-theoretic analysis of data selection and learning in pool-based AL. Applying finite blocklength analysis of noisy lossy compression, we derive information-theoretic lower bounds on label complexity and generalization error that serve as theoretical limits for a given learning algorithm under its associated optimal data selection strategy. Specifically, our bounds include terms that reflect overfitting induced by the learning algorithm and the discrepancy between its inductive bias and the target task, and are closely related to established information-theoretic bounds and stability theory, which have not been previously applied to the analysis of pool-based AL. These properties yield a new theoretical perspective on pool-based AL.

💡 Research Summary

The paper introduces an information‑theoretic framework that reinterprets pool‑based active learning (AL) as a noisy lossy compression problem, thereby enabling the use of finite‑blocklength analysis to derive fundamental lower bounds on label complexity and generalization error. Traditional pool‑based AL theory has largely focused on hypothesis‑set complexity, disagreement coefficients, or distributional properties to characterize the number of labels needed for a target error. Such approaches, however, do not incorporate algorithm‑specific factors such as the degree of overfitting or the mismatch between an algorithm’s inductive bias and the true data‑generating distribution—factors that are central to modern learning theory (e.g., information‑theoretic bounds, stability theory, PAC‑Bayes).

Building on the earlier work of Sugiyama & Uchida (2026), which mapped the learning process to fixed‑length lossy compression (training data ↔ codewords, sampling ↔ encoder, learning ↔ decoder), the authors observe that this prior framework assumes unconstrained sampling from the source distribution. Consequently, it cannot capture the essential restriction of pool‑based AL, where the learner must select a subset from a pre‑existing finite pool. To address this, the paper adopts the “noisy lossy compression” model of Kostina & Verdú (2016). In this model a source symbol A passes through a channel c, producing a noisy observation A′, after which an encoder f compresses A′. The authors map the channel to the pool observation step (the pool is a noisy version of the underlying source distribution) and the encoder to the data‑selection strategy (choosing which pool items to label). The decoder corresponds to the learning algorithm that produces a hypothesis H.

With this correspondence, the authors apply the finite‑blocklength results of Kostina & Verdú to pool‑based AL. The key quantity becomes the mutual information I(W;H) between the latent sub‑distribution variable W (which indexes the regions of the input space) and the learned hypothesis H. Minimizing I(W;H) under a distortion constraint (the desired generalization error) yields a lower bound on the number of labels n (or equivalently the selection rate R = bn/k, where b is the bits needed to encode a single example). The bound decomposes into two interpretable terms:

1. Overfitting term – captures how much information about the training set the algorithm memorizes. It appears in forms familiar from information‑theoretic generalization bounds (e.g., KL divergence between posterior and prior) and from algorithmic stability analyses.
2. Inductive‑bias mismatch term – measures the divergence between the algorithm’s implicit prior (or hypothesis class) and the true data distribution, reflecting how well the algorithm’s bias aligns with the task.

Both terms are weighted by factors that depend on the pool size m and the number of selected samples n, thereby explicitly incorporating the pool‑based constraint that is absent in prior work. The authors also treat the special case where the entire pool is labeled (n = m), showing that the bound reduces to known i.i.d. sampling limits, confirming consistency with existing theory.

The paper discusses how these lower bounds compare with common active‑learning heuristics (uncertainty sampling, core‑set selection, etc.). While the bounds are generally loose—reflecting worst‑case information‑theoretic limits—they become tighter when the learning algorithm exhibits significant overfitting or when its inductive bias is poorly matched to the task, situations where traditional hypothesis‑set based bounds may be vacuous. Thus, the framework provides a novel lens that links label complexity directly to algorithmic properties rather than solely to hypothesis‑set geometry.

In summary, the contribution consists of (i) establishing a rigorous mapping from pool‑based AL to noisy lossy compression, (ii) deriving finite‑blocklength lower bounds on label complexity and generalization error that explicitly account for pool constraints, and (iii) revealing how overfitting and bias‑mismatch terms naturally arise within these bounds. This bridges a gap between information‑theoretic learning theory, stability analysis, and active learning, offering a unified perspective that could guide the design of more principled active‑learning strategies and provide benchmark limits for evaluating their performance.