Taming the Monster Every Context: Complexity Measure and Unified Framework for Offline-Oracle Efficient Contextual Bandits

We propose an algorithmic framework, Offline Estimation to Decisions (OE2D), that reduces contextual bandit learning with general reward function approximation to offline regression. The framework allows near-optimal regret for contextual bandits with large action spaces with $O(log(T))$ calls to an offline regression oracle over $T$ rounds, and makes $O(loglog(T))$ calls when $T$ is known. The design of OE2D algorithm generalizes Falcon~\citep{simchi2022bypassing} and its linear reward version~\citep[][Section 4]{xu2020upper} in that it chooses an action distribution that we term ``exploitative F-design’’ that simultaneously guarantees low regret and good coverage that trades off exploration and exploitation. Central to our regret analysis is a new complexity measure, the Decision-Offline Estimation Coefficient (DOEC), which we show is bounded in bounded Eluder dimension per-context and smoothed regret settings. We also establish a relationship between DOEC and Decision Estimation Coefficient (DEC)~\citep{foster2021statistical}, bridging the design principles of offline- and online-oracle efficient contextual bandit algorithms for the first time.

💡 Research Summary

The paper introduces OE2D (Offline‑Estimation‑to‑Decision), a unified algorithmic framework that reduces contextual bandit learning with general reward function approximation to offline regression. Unlike prior approaches that either rely on online regression or make O(T) calls to an offline oracle, OE2D achieves near‑optimal √T‑type regret while calling the offline regression oracle only O(log T) times (or O(log log T) when the horizon T is known). The core innovation is the “exploitative F‑design” distribution, computed for each context, which simultaneously satisfies a low‑regret condition (the expected reward under the distribution is close to optimal) and a good‑coverage condition (the distribution provides sufficient information for accurate offline regression). This design generalizes the pure‑exploration F‑design to incorporate exploitation, enabling efficient exploration in large or continuous action spaces.

A new complexity measure, the Decision‑Offline Estimation Coefficient (DOEC), is defined as the worst‑case ratio between the offline regression error and the instantaneous regret for a given exploration distribution. DOEC differs from the previously studied Decision Estimation Coefficient (DEC) by not referencing a central model, making it directly applicable to offline estimation. The authors prove structural results linking DOEC to a modified Sequential Extrapolation Coefficient (ε‑SEC), showing that DOEC captures “active” experimental design power beyond the “passive” ε‑SEC. Consequently, in settings with bounded per‑context Eluder dimension or h‑smoothed regret, DOEC remains constant, yielding the desired logarithmic oracle‑call complexity.

Theoretical contributions include: (1) Regret guarantees (Theorem 1) that recover and extend existing results for discrete actions, linear rewards, and per‑context linear models; (2) A modular analysis (Theorem 2) that handles model misspecification, reward corruption, and distribution shift; (3) A relationship between DOEC and DEC (Theorem 5) establishing that any exploration distribution with small DOEC also yields small DEC, thereby bridging offline‑ and online‑oracle efficient designs for the first time.

Empirically, Table 1 compares OE2D with prior algorithms, highlighting that OE2D attains comparable or better regret while dramatically reducing oracle calls. The framework is flexible: when T is known, a specific schedule reduces calls to O(log log T). Overall, the paper provides a comprehensive, statistically grounded, and computationally efficient solution to contextual bandits with general function approximation, unifying and extending the design principles of both offline and online oracle‑efficient methods.