Inference for Batched Adaptive Experiments

The advantages of adaptive experiments have led to their rapid adoption in economics, other fields, as well as among practitioners. However, adaptive experiments pose challenges for causal inference. This note suggests a BOLS (batched ordinary least squares) test statistic for inference of treatment effects in adaptive experiments. The statistic provides a precision-equalizing aggregation of per-period treatment-control differences under heteroskedasticity. The combined test statistic is a normalized average of heteroskedastic per-period z-statistics and can be used to construct asymptotically valid confidence intervals. We provide simulation results comparing rejection rates in the typical case with few treatment periods and few (or many) observations per batch.

💡 Research Summary

The paper addresses a central methodological challenge in adaptive experiments: how to conduct valid causal inference when treatment assignment probabilities evolve over time based on past outcomes and when outcome variances differ across treatment arms (heteroskedasticity). In the standard randomized controlled trial, each observation is independent and identically assigned, allowing simple ordinary least‑squares (OLS) or difference‑in‑means estimators to be treated as asymptotically normal. In adaptive settings, however, the assignment probability πₜ in batch t is a random variable that depends on the experiment’s history, and the variances σ²₁,ₜ and σ²₀,ₜ of the treated and control outcomes may differ. This dual source of randomness can render the usual OLS estimator non‑normal and cause confidence intervals to have incorrect coverage.

The authors propose a “batched ordinary least squares” (BOLS) statistic that aggregates the per‑batch treatment‑control differences using precision‑weighting. For each batch t, they define the sample‑mean difference (\hat\Delta_t = \bar Y_{1,t} - \bar Y_{0,t}) and its variance
(v_t = \sigma_{1,t}^2/N_{1,t} + \sigma_{0,t}^2/N_{0,t} = \frac{1}{n_t}\bigl(\sigma_{1,t}^2\pi_t + \sigma_{0,t}^2(1-\pi_t)\bigr)).
The weight is set to (w_t = 1/\sqrt{v_t}). The overall estimator is the weighted average
\