Beyond ATE: Multi-Criteria Design for A/B Testing

In the era of large-scale AI deployment and high-stakes clinical trials, adaptive experimentation faces a trilemma'' of conflicting objectives: minimizing cumulative regret (welfare loss during the experiment), maximizing the estimation accuracy of heterogeneous treatment effects (CATE), and ensuring differential privacy (DP) for participants. Existing literature typically optimizes these metrics in isolation or under restrictive parametric assumptions. In this work, we study the multi-objective design of adaptive experiments in a general non-parametric setting. First, we rigorously characterize the instance-dependent Pareto frontier between cumulative regret and estimation error, revealing the fundamental statistical limits of dual-objective optimization. We propose ConSE, a sequential segmentation and elimination algorithm that adaptively discretizes the covariate space to achieve the Pareto-optimal frontier. Second, we introduce DP-ConSE, a privacy-preserving extension that satisfies Joint Differential Privacy. We demonstrate that privacy comes for free’’ in our framework, incurring only asymptotically negligible costs to regret and estimation accuracy. Finally, we establish a robust link between experimental design and long-term utility: we prove that any policy derived from our Pareto-optimal algorithms minimizes post-experiment simple regret, regardless of the specific exploration-exploitation trade-off chosen during the trial. Our results provide a theoretical foundation for designing ethical, private, and efficient adaptive experiments in sensitive domains.

💡 Research Summary

The paper tackles a newly articulated “trilemma” in adaptive experimentation: minimizing cumulative regret (the welfare loss incurred during the trial), maximizing the accuracy of heterogeneous treatment effect (CATE) estimation, and guaranteeing differential privacy (DP) for participants. While prior work has typically treated these objectives in isolation—bandit literature focusing on regret, Neyman‑style designs on average treatment effect estimation, and privacy research on static data—this work unifies them in a non‑parametric contextual bandit setting.

First, the authors derive an instance‑specific information‑theoretic lower bound that characterizes the Pareto frontier between cumulative regret and CATE mean‑squared error under a given regret budget. The bound depends on the smoothness (Lipschitz) of the reward functions and a margin condition that captures how quickly the treatment advantage vanishes near decision boundaries. This result shows that for any algorithm, the achievable estimation error cannot be smaller than a function of the allowed regret and the instance’s margin parameter.

Building on this characterization, the paper introduces ConSE (Sequential Segmentation and Elimination). ConSE adaptively partitions the covariate space into a hierarchical grid. At each round it (i) estimates the treatment advantage in each cell, (ii) refines cells whose uncertainty exceeds a threshold, and (iii) eliminates cells where one arm is statistically dominant, thereby reallocating samples to more informative regions. A single hyper‑parameter α∈