Positive Distribution Shift as a Framework for Understanding Tractable Learning

We study a setting where the goal is to learn a target function f(x) with respect to a target distribution D(x), but training is done on i.i.d. samples from a different training distribution D’(x), labeled by the true target f(x). Such a distribution shift (here in the form of covariate shift) is usually viewed negatively, as hurting or making learning harder, and the traditional distribution shift literature is mostly concerned with limiting or avoiding this negative effect. In contrast, we argue that with a well-chosen D’(x), the shift can be positive and make learning easier – a perspective called Positive Distribution Shift (PDS). Such a perspective is central to contemporary machine learning, where much of the innovation is in finding good training distributions D’(x), rather than changing the training algorithm. We further argue that the benefit is often computational rather than statistical, and that PDS allows computationally hard problems to become tractable even using standard gradient-based training. We formalize different variants of PDS, show how certain hard classes are easily learnable under PDS, and make connections with membership query learning.

💡 Research Summary

The paper introduces a new perspective on covariate shift, arguing that a carefully chosen training distribution D′, different from the target distribution D, can actually facilitate learning rather than hinder it. This phenomenon is termed Positive Distribution Shift (PDS). The authors formalize PDS by defining a learning rule A that, given m(ε) i.i.d. samples from (D′, f), produces a hypothesis whose error on the true joint distribution (D, f) is within ε of the best possible error achievable by any hypothesis in the class H, while running in time T(ε). The definition deliberately leaves the quantifiers over A and D′ open, allowing later sections to instantiate concrete variants.

The central claim is that PDS mainly yields computational benefits: many learning problems that are information‑theoretically easy but computationally intractable under the standard PAC setting become tractable when the training distribution is altered, even though the target distribution remains unchanged. The authors illustrate this claim through several canonical hard classes.

Parity functions provide the first, detailed case study. Under the uniform distribution, k‑sparse parity functions are statistically easy (O(k log d) samples suffice) but believed to be computationally hard for any polynomial‑time algorithm, especially in the presence of label noise. The paper shows that if the training distribution D′ biases each input bit slightly (e.g., each coordinate has a small non‑zero mean), then coordinates belonging to the parity’s support exhibit a noticeably larger correlation with the label than irrelevant coordinates. This structural asymmetry can be exploited by a simple correlation test or by gradient descent on a modest neural network, yielding a polynomial‑time algorithm that recovers the support and thus learns the parity. The authors prove a formal theorem (Theorem 4.3) establishing a tractable PDS algorithm for any parity, and they further demonstrate that standard stochastic gradient descent (SGD) on a two‑layer ReLU network empirically learns both sparse and dense parities when trained on the biased distribution and evaluated on the uniform test distribution.

Moving beyond data‑independent PDS, the authors define function‑dependent PDS (f‑PDS), where the training distribution may be tailored to the specific target function f. In this generous model, they prove (Theorem 3.2) that any Boolean circuit of size s, or any neural network of size s with polylog‑precision weights, can be learned with polynomial sample and time complexity by SGD on a non‑standard feed‑forward network (special topology or initialization). The construction essentially encodes bits of f into subtle statistical properties of D′, allowing the network to “decode” the target without relying on the labels. While this shows the theoretical power of f‑PDS, the reliance on non‑standard architectures means it does not directly address the practical question of whether standard networks can achieve the same.

To bridge this gap, the paper proposes a DS‑PAC framework, restricting D′ to depend only on the hypothesis class H and the target distribution D, but not on the specific function f. Within DS‑PAC, the authors present algorithms for learning parities, depth‑bounded Boolean circuits, and noisy label settings, all with polynomial sample and runtime guarantees. The key idea is to design D′ that amplifies certain structural features (e.g., biasing variables that appear frequently in the target class) while preserving the test distribution D for evaluation.

The experimental section validates the theory. Using standard SGD on a two‑layer ReLU network, the authors train on biased input distributions and test on the original uniform distribution. Results show rapid convergence and low test error for both parity functions (including dense parities) and randomly generated Boolean circuits of moderate size. These experiments suggest that the positive effect of distribution shift observed in theory can be realized with off‑the‑shelf deep‑learning pipelines.

In summary, the paper makes several contributions:

1. Conceptual shift: reframes covariate shift as a potentially positive tool rather than a nuisance.
2. Formal framework: defines PDS, f‑PDS, and DS‑PAC, clarifying the roles of the training distribution and learning algorithm.
3. Theoretical results: demonstrates polynomial‑time learnability of classes previously considered computationally hard (parities, certain circuits) under appropriate PDS.
4. Algorithmic insights: shows that simple correlation‑based methods or standard SGD on modest neural nets can exploit the bias introduced by D′.
5. Empirical evidence: provides experiments confirming that standard deep‑learning training can benefit from a well‑chosen training distribution.

The work opens several avenues for future research: (i) characterizing the minimal conditions on D′ that guarantee tractability for broader function families; (ii) developing automated methods (e.g., meta‑learning or curriculum design) to construct beneficial training distributions in real‑world settings; and (iii) analyzing how PDS interacts with generalization, robustness, and fairness when the test distribution is fixed but the training distribution is deliberately altered. Overall, the paper argues convincingly that “designing the data” can be as powerful as “designing the algorithm” for achieving tractable learning.