Approximate Bayesian Computation with Statistical Distances for Model Selection

Model selection in the presence of intractable likelihoods remains a central challenge in Bayesian inference. Approximate Bayesian computation (ABC) provides a flexible likelihood-free framework, but its use for model choice is known to be sensitive to the choice of summary statistics, often leading to poorly calibrated posterior model probabilities. Recent ABC variants based on statistical distances allow comparisons to be performed directly on empirical distributions, avoiding data reduction and offering improved theoretical guarantees under suitable conditions. This paper provides a systematic evaluation of discrepancy-based ABC methods for Bayesian model selection, focusing on their empirical behavior across a range of simulation settings and levels of model complexity. We compare full data ABC approaches based on Wasserstein, Creamer-von-Mises, and maximum mean discrepancy metrics with summary-statistic-based ABC and neural network classifiers. The results highlight settings in which full data ABC yields stable and well-calibrated posterior model probabilities, as well as scenarios where performance degrades due to model overlap or dependence. An application to toad movement models illustrates the practical implications of these findings. Overall, the study clarifies the strengths and limitations of discrepancy-based ABC for likelihood-free model choice and provides guidance for its use in realistic inferential settings.

💡 Research Summary

This paper addresses a central difficulty in Bayesian model selection when likelihood functions are intractable: the sensitivity of Approximate Bayesian Computation (ABC) to the choice of low‑dimensional summary statistics. Traditional ABC reduces high‑dimensional data to a few informative summaries and then accepts simulated datasets whose summaries lie within a tolerance ε of the observed summaries. While this works reasonably for parameter inference, it often yields poorly calibrated posterior model probabilities for model choice, especially when models overlap or when sufficient statistics are unavailable.

To overcome these limitations, the authors propose a “full‑data” ABC framework that operates directly on empirical distributions of the observed and simulated data, bypassing any summary reduction. They explore three integral probability semimetrics as discrepancy measures: the 1‑Wasserstein distance, the Cramér–von Mises (CvM) distance, and the Maximum Mean Discrepancy (MMD) with a Gaussian kernel. For each metric, unbiased empirical estimators are derived (order‑statistic differences for Wasserstein, rank‑based formula for CvM, and U‑statistic for MMD). The acceptance rule is unchanged: a simulated dataset is kept if the chosen distance between the two empirical measures is ≤ ε.

Theoretical contributions build on recent work by Legraman et al. (2025), which provides uniform concentration bounds for ABC posteriors under integral probability semimetrics. The paper shows that, when ε_n → 0 as the sample size n grows, the Wasserstein‑based ABC (ABC‑Wass) yields consistent parameter estimates and, under a well‑separated‑model condition (the true and competing models generate distinguishable empirical distributions in Wasserstein sense), the resulting posterior model probabilities converge to the true Bayes factors. The authors discuss the necessity of decreasing ε with n, the role of i.i.d. assumptions, and note that extensions to dependent data are possible but left for future work.

A comprehensive simulation study evaluates the performance of full‑data ABC against two baselines: (i) classic summary‑statistic ABC with various handcrafted summaries, and (ii) neural‑network classifiers trained to approximate posterior model probabilities (NN‑ABC). Four scenarios are considered: (a) simple Gaussian models, (b) mixtures of Gaussians, (c) autoregressive time‑series models, and (d) high‑dimensional multivariate normals. For each scenario, three candidate models are compared. Performance metrics include calibration (via probability integral transform), classification accuracy, and mean absolute error of posterior probabilities. Results show that when models are well separated (scenarios a and b), full‑data ABC with any of the three distances outperforms both baselines, with Wasserstein and MMD delivering the most stable calibration. In more challenging settings (scenarios c and d), all methods degrade, but NN‑ABC tends to overfit, producing overly confident but biased posterior probabilities, while summary‑based ABC suffers heavily from poor summary choice.

The methodology is applied to a real ecological case: toad movement models originally described by Marchand et al. (2017). Two competing mechanistic models differ in how they treat step length and turning angle distributions. Using a set of handcrafted summaries (mean step length, mean turning angle) leads to unstable model probabilities. In contrast, full‑data ABC based on Wasserstein and MMD distances, which compare the entire trajectory point clouds, yields well‑calibrated posterior probabilities that clearly favor the model better supported by the data. This empirical example demonstrates that full‑data ABC can be employed in practice without the arduous task of designing sufficient summaries.

The paper concludes with practical recommendations: (1) choose a discrepancy metric aligned with the scientific question (Wasserstein is sensitive to quantile differences, MMD captures higher‑order moment discrepancies, CvM offers robustness); (2) set the tolerance ε adaptively, decreasing with sample size to guarantee asymptotic consistency; (3) be aware that model separation in the chosen metric space is essential for reliable model selection; (4) acknowledge that extensions to non‑i.i.d. or high‑dimensional dependent data require further theoretical development. All code and simulation scripts are made publicly available, facilitating reproducibility and encouraging future research on likelihood‑free Bayesian model selection using full‑data discrepancy measures.