Wasserstein-enabled characterization of designs and myopic decisions in Bayesian Optimization

Impractical assumptions, an inherently myopic nature, and the crucial role of the initial design, all together contribute to making theoretical convergence proofs of little value in real-life Bayesian Optimization applications. In this paper, we propose a novel characterization of the design depending on its distributional properties, separately measured with respect to the coverage of the search space and the concentration around the best observed function value. These measures are based on the Wasserstein distance and enable a model-free evaluation of the information value of the design before deciding the next query. Then, embracing the myopic nature of Bayesian Optimization, we take an empirical approach to analyze the relation between the proposed characterization of the design and the quality of the next query. Ultimately, we provide important and useful insights that might inspire the definition of a new generation of acquisition functions in Bayesian Optimization.

💡 Research Summary

Bayesian Optimization (BO) is a powerful sequential model‑based approach for global optimization of expensive black‑box functions, but its theoretical convergence guarantees rely on unrealistic assumptions such as exact knowledge of the kernel’s hyper‑parameters and the function belonging to a specific reproducing kernel Hilbert space. In practice, model misspecification, hyper‑parameter estimation, and especially the quality of the initial design (the set of observed points) heavily influence whether BO converges to the global optimum or gets trapped in local minima.
To address this gap, the authors propose a model‑free, distribution‑based characterization of a design D = (X, Y) using two Wasserstein‑based metrics. The first metric,
 S₁(D) = W₂²(X, G_X),
measures the 2‑Wasserstein distance between the empirical distribution of the sampled locations X and a uniform grid G_X covering the search space. A small S₁ indicates that the points are uniformly spread, i.e., the design provides good coverage. The second metric,
 S₂(D) = W₂²(Y, δ_{y⁺}),
measures the distance between the empirical distribution of observed function values Y and a Dirac delta located at the current best value y⁺ = min Y. This captures not only the variance of Y but also how tightly the observations concentrate around the best value, something a simple standard deviation cannot express.
Together, (S₁, S₂) embed any design into a two‑dimensional “information‑value” space that can be computed before the next query, without any reference to the surrogate model.
The authors empirically investigate how these metrics relate to the quality of the next query x′, defined by the improvement Δy = y⁺ − y′ (positive Δy means improvement). Experiments are conducted on 14 benchmark functions (8 one‑dimensional, 6 two‑dimensional). Three families of designs are generated: (1) pure Latin Hypercube Sampling (LHS), (2) LHS mixed with points drawn from a neighbourhood of the true optimum (simulating a BO process that is converging correctly), and (3) LHS mixed with points from a neighbourhood of a random sub‑optimal point (simulating a BO process that is converging to the wrong region). For each design size n ∈ {5d, ⌊12.5d⌋, 20d} a Gaussian Process with a Matérn 3/2 kernel is fitted via maximum‑likelihood, and four common acquisition functions are evaluated: Surface‑Response (SR), Standard‑Deviation maximization (SD), Expected Improvement (EI), and Lower Confidence Bound (LCB). The acquisition is optimized on a dense grid of 10 000 points to avoid stochastic optimisation effects.
Key findings:

• S₁ strongly decreases with n and is lowest for pure LHS designs, confirming that larger designs provide better space coverage. S₁ is largely independent of the underlying objective function.
• S₂ does not correlate with n or design type; instead it reflects the shape of the objective. Highly multimodal or oscillatory functions yield large S₂ values because observations are spread over many local optima.
• Designs that simultaneously achieve low S₁ (good coverage) and low S₂ (observations tightly clustered around the current best) tend to produce positive Δy across all acquisition functions. In other words, the (S₁, S₂) location predicts the likelihood that the next query will improve the incumbent.
• Model misspecification, measured by the root‑mean‑square error (RMSE) between the GP posterior mean and the true function on the grid, does not systematically degrade Δy. High RMSE can coexist with large improvements, indicating that the distributional properties of the design are a more direct predictor of BO performance than the surrogate’s global fit.
  These results suggest that a pre‑query, model‑free assessment of the design using Wasserstein distances can guide the selection or adaptation of acquisition strategies. For instance, when a design exhibits high S₁ but also high S₂ (good coverage but observations scattered), an acquisition function that emphasizes exploration may be preferable; conversely, low S₁ and low S₂ indicate that the optimizer is already focusing on a promising region, and a more exploitative acquisition could be used.
  Overall, the paper contributes a principled statistical tool to quantify the information content of a BO design, demonstrates empirically that this quantification correlates with the quality of myopic decisions, and opens avenues for next‑generation acquisition functions that explicitly incorporate Wasserstein‑based design diagnostics.