Certificate-Guided Pruning for Stochastic Lipschitz Optimization

We study black-box optimization of Lipschitz functions under noisy evaluations. Existing adaptive discretization methods implicitly avoid suboptimal regions but do not provide explicit certificates of optimality or measurable progress guarantees. We introduce \textbf{Certificate-Guided Pruning (CGP)}, which maintains an explicit \emph{active set} $A_t$ of potentially optimal points via confidence-adjusted Lipschitz envelopes. Any point outside $A_t$ is certifiably suboptimal with high probability, and under a margin condition with near-optimality dimension $α$, we prove $\Vol(A_t)$ shrinks at a controlled rate yielding sample complexity $\tildeO(\varepsilon^{-(2+α)})$. We develop three extensions: CGP-Adaptive learns $L$ online with $O(\log T)$ overhead; CGP-TR scales to $d > 50$ via trust regions with local certificates; and CGP-Hybrid switches to GP refinement when local smoothness is detected. Experiments on 12 benchmarks ($d \in [2, 100]$) show CGP variants match or exceed strong baselines while providing principled stopping criteria via certificate volume.

💡 Research Summary

The paper tackles stochastic Lipschitz optimization, where the objective function f is only accessible through noisy point evaluations and satisfies a global Lipschitz condition |f(x)−f(y)| ≤ L·d(x,y). Classical approaches such as DIRECT, Lipschitz bandits, and adaptive discretization implicitly avoid sub‑optimal regions but never expose an explicit certificate that a region can be safely discarded, nor do they provide a measurable progress indicator. The authors introduce Certificate‑Guided Pruning (CGP), a novel algorithm that maintains an explicit “active set” Aₜ of potentially optimal points at each iteration.

CGP builds on confidence intervals for each sampled point xᵢ. With sub‑Gaussian noise of variance σ², the empirical mean \hat μᵢ(t) and confidence radius rᵢ(t)=σ√(2 log(2 Nₜ T/δ)/nᵢ) are computed, yielding an upper confidence bound UCBᵢ(t)=\hat μᵢ+rᵢ and a lower confidence bound LCBᵢ(t)=\hat μᵢ−rᵢ. The global Lipschitz upper envelope is defined as

Uₜ(x)=min_i { UCBᵢ(t) + L·d(x,xᵢ) },

and the global lower certificate as

ℓₜ = max_i LCBᵢ(t).

The active set is then

Aₜ = { x ∈ X : Uₜ(x) ≥ ℓₜ }.

Any point outside Aₜ has an upper bound strictly below the current lower bound on the optimum, so it is certifiably sub‑optimal with probability at least 1−δ. The algorithm selects the next query point by maximizing a score that balances exploitation (high Uₜ) and exploration (distance to existing samples), namely

xₜ₊₁ = argmax_{x∈Aₜ}