A tutorial on conformal prediction

Conformal prediction uses past experience to determine precise levels of confidence in new predictions. Given an error probability $\epsilon$, together with a method that makes a prediction $\hat{y}$ of a label $y$, it produces a set of labels, typically containing $\hat{y}$, that also contains $y$ with probability $1-\epsilon$. Conformal prediction can be applied to any method for producing $\hat{y}$: a nearest-neighbor method, a support-vector machine, ridge regression, etc. Conformal prediction is designed for an on-line setting in which labels are predicted successively, each one being revealed before the next is predicted. The most novel and valuable feature of conformal prediction is that if the successive examples are sampled independently from the same distribution, then the successive predictions will be right $1-\epsilon$ of the time, even though they are based on an accumulating dataset rather than on independent datasets. In addition to the model under which successive examples are sampled independently, other on-line compression models can also use conformal prediction. The widely used Gaussian linear model is one of these. This tutorial presents a self-contained account of the theory of conformal prediction and works through several numerical examples. A more comprehensive treatment of the topic is provided in “Algorithmic Learning in a Random World”, by Vladimir Vovk, Alex Gammerman, and Glenn Shafer (Springer, 2005).

💡 Research Summary

The paper “A Tutorial on Conformal Prediction” provides a self‑contained exposition of conformal prediction, a methodology that turns any point‑prediction algorithm into a set‑valued predictor with rigorous, finite‑sample guarantees. The central idea is to attach a nonconformity measure A to each example, quantifying how “strange” a candidate label looks relative to the previously observed data. For a new object xₙ, the algorithm computes the nonconformity scores of all possible labels (or a dense grid in regression), ranks them, and derives p‑values by comparing the score of each candidate with the scores of the training examples. For a pre‑specified error level ε, the prediction region Γ_ε consists of all labels whose p‑value exceeds ε. By construction, under the assumption of exchangeability (a very weak condition that includes i.i.d. as a special case), the probability that the true label falls outside Γ_ε is at most ε, regardless of the underlying distribution or the choice of the underlying point predictor.

The authors emphasize the online nature of conformal prediction: after each prediction the true label is revealed, the data set grows, and the next prediction is made using the enlarged data set. Remarkably, the “hit” events—instances where the true label lies inside the prediction region—are probabilistically independent across time, even though the data sets overlap. This independence yields a law‑of‑large‑numbers effect: over many steps the empirical error rate converges to the nominal ε. The paper contrasts this with classical confidence intervals, which are usually justified only for a single, fixed data set. Conformal prediction therefore extends the notion of confidence to a sequential, adaptive learning scenario.

A substantial portion of the tutorial is devoted to illustrating the theory with concrete examples. The authors revisit Fisher’s classic 95 % prediction interval for a normal mean, showing how the same interval can be derived as a conformal prediction region using the residual‑based nonconformity measure. They work through a historical data set (Czuber’s student counts) to demonstrate the calculations step‑by‑step, and they explain why the resulting intervals remain valid in the online setting. The paper also discusses how the same ideas apply to linear regression with Gaussian errors, where the residuals again serve as a natural nonconformity score, and the resulting intervals coincide with textbook t‑distribution intervals.

Beyond the normal model, the authors introduce the concept of online compression models, a broad class that includes the Gaussian linear model as a special case. In these models, the data are summarized by a “compression” (e.g., sufficient statistics), and the conformal algorithm works with the compressed representation. This perspective shows that conformal prediction is not limited to exchangeable sequences; it can be adapted to other structured stochastic processes, provided a suitable compression scheme and nonconformity measure are defined.

Efficiency—how small the prediction regions are—is another major theme. While validity is guaranteed under minimal assumptions, the size of Γ_ε depends heavily on the choice of the nonconformity measure and on any prior knowledge about the data‑generating distribution Q. If the practitioner has a good model of Q, they can design a nonconformity function that yields tight regions (e.g., using Bayesian posterior means as point predictors and residuals as scores). Conversely, a overly conservative nonconformity measure leads to large, uninformative regions that may contain all possible labels. The tutorial therefore encourages a two‑step approach: first secure validity with a generic nonconformity measure, then refine it using domain knowledge to improve efficiency.

The paper also addresses the philosophical debate between classical confidence intervals and full Bayesian conditional probabilities. Conformal prediction occupies a middle ground: before seeing any data it offers a pre‑data guarantee (the 1‑ε coverage), yet after observing data it does not claim a posterior probability for the specific region—it simply reports the set. This avoids the pitfalls of over‑interpreting a single realized interval while still providing a useful, objective measure of uncertainty.

In the concluding sections, the authors point readers to the more extensive treatment in “Algorithmic Learning in a Random World” and to related work on randomness, game‑theoretic probability, and extensions beyond the online setting. The tutorial, however, stands on its own as a practical guide: Sections 4.2 and 4.3 give algorithmic pseudocode, and the numerical examples illustrate how to implement conformal predictors for classification (including the case of only two possible labels) and regression.

Overall, the paper demonstrates that conformal prediction is a powerful, model‑agnostic tool for quantifying uncertainty in machine‑learning predictions. It delivers finite‑sample, distribution‑free coverage guarantees in an online, sequential learning environment, while allowing practitioners to tailor the method for efficiency using problem‑specific nonconformity measures. This makes it highly relevant for real‑world applications where reliable confidence statements are essential, such as medical diagnosis, financial risk assessment, and autonomous systems.