Mixture priors for replication studies

Replication of scientific studies is important for assessing the credibility of their results. However, there is no consensus on how to quantify the extent to which a replication study replicates an original result. We propose a novel Bayesian approach for replication studies based on mixture priors. The idea is to use a mixture of the posterior distribution based on the original study and a non-informative distribution as the prior for the analysis of the replication study. The mixture weight then determines the extent to which the original and replication data are pooled. Two distinct strategies are presented: one with fixed mixture weights, and one that introduces uncertainty by assigning a prior distribution to the mixture weight itself. Furthermore, it is shown how within this framework Bayes factors can be used for formal testing of relevant scientific hypotheses, such as tests on the presence or absence of an effect or whether the mixture weight equals zero (completely discounting the original data) or one (fully pooling with the original data). To showcase the practical application of the methodology, we analyze data from three replication studies. Our findings suggest that mixture priors are a valuable and intuitive alternative to other Bayesian methods for analyzing replication studies, such as hierarchical models and power priors. We provide the free and open source R package repmix that implements the proposed methodology.

💡 Research Summary

**
The paper introduces a novel Bayesian framework for analyzing replication studies that leverages mixture priors to balance information from the original study with a non‑informative baseline. The authors propose constructing a prior for the replication analysis as a weighted mixture of the original study’s posterior distribution (centered at the original effect estimate θ̂₀ with variance σ₀²) and a diffuse normal prior N(μ,τ²) that represents minimal borrowing. The mixture weight ω (0 ≤ ω ≤ 1) directly controls the degree of pooling: ω = 1 corresponds to full pooling (the original data are treated as if they were part of the replication), while ω = 0 discards the original information entirely.

Two strategies for handling ω are presented. The first fixes ω at a pre‑specified value (e.g., 0.5, 0.8), allowing researchers to encode prior beliefs about the similarity of the studies. The second treats ω as an unknown parameter and assigns it a Beta(a,b) prior, letting the data determine how much borrowing is appropriate. This second approach yields a posterior distribution for ω, providing a direct measure of agreement between the original and replication studies.

The likelihood for the replication data is modeled as normal: each replication effect estimate θ̂ᵣᵢ is assumed to follow N(θ,σᵣᵢ²), where σᵣᵢ is the known standard error. For multiple replications the authors combine the individual estimates into a pooled estimate θ̂ᵣₚ with variance σᵣₚ² using inverse‑variance weighting, which simplifies the analysis to a single normal likelihood for θ.

Bayes factors are derived for several scientifically relevant hypotheses: (i) the presence of an effect (θ = 0 vs θ ≠ 0), (ii) whether the original data should be ignored (ω = 0), and (iii) whether the original and replication data are fully compatible (ω = 1). Posterior predictive checks are also employed to assess model fit.

The methodology is illustrated with three direct replications of the “moral credentialing” experiment from Many Labs 3. The Toronto and Montana State replications yielded posterior ω estimates around 0.7–0.8, indicating substantial borrowing and consistency with the original effect (θ̂₀ = 0.21). In contrast, the Ashland replication produced ω̂ ≈ 0.15, reflecting strong disagreement and even a sign reversal of the effect. Bayes factors corroborated these conclusions, strongly favoring the hypothesis of no borrowing for the Ashland case.

A comparative discussion positions mixture priors against hierarchical models and power priors. Hierarchical models introduce a between‑study variance τ², but interpreting τ² as a borrowing degree can be opaque. Power priors raise the original likelihood to a power α, yet α is typically fixed, offering limited flexibility. In contrast, the mixture‑prior framework provides an explicit, interpretable weight ω and, when modeled with a Beta prior, captures uncertainty about borrowing directly.

The authors provide an open‑source R package, repmix, which implements fixed‑ω and random‑ω analyses, Bayes factor computation, and posterior predictive diagnostics, making the approach readily applicable to a wide range of replication contexts.

Limitations include the reliance on normal approximations for effect estimates, which may be problematic for small samples or non‑continuous outcomes, and sensitivity to the choice of the non‑informative prior’s parameters (μ, τ²). The paper suggests extensions such as non‑normal likelihoods, multi‑level mixture priors for several original studies, and empirical Bayes estimation of the Beta hyper‑parameters to reduce subjectivity.

In sum, the mixture‑prior approach offers a conceptually simple yet powerful tool for quantifying replication success, allowing researchers to explicitly control and assess the influence of historical data, to test scientifically meaningful hypotheses via Bayes factors, and to implement the method easily through the provided software.