On The Statistical Limits of Self-Improving Agents

As systems trend toward superintelligence, a natural modeling premise is that agents can self-improve along every facet of their own design. We formalize this with a five-axis decomposition and a decision layer, separating incentives from learning behavior and analyzing axes in isolation. Our central result identifies and introduces a sharp utility-learning tension, the structural conflict in self-modifying systems whereby utility-driven changes that improve immediate or expected performance can also erode the statistical preconditions for reliable learning and generalization. Our findings show that distribution-free guarantees are preserved iff the policy-reachable model family is uniformly capacity-bounded; when capacity can grow without limit, utility-rational self-changes can render learnable tasks unlearnable. Under standard assumptions common in practice, these axes reduce to the same capacity criterion, yielding a single boundary for safe self-modification.

💡 Research Summary

The paper tackles a fundamental gap in learning theory: the assumption that the learning mechanism—its algorithm, representation, architecture, computational substrate, and meta‑cognitive control—remains fixed. As artificial agents move toward superintelligence, this assumption becomes unrealistic; agents are expected to rewrite large portions of their own design. To study the consequences of such self‑modification, the authors introduce a five‑axis decomposition (Algorithmic A, Representational H, Architectural Z, Substrate F, Metacognitive M) together with a decision‑layer that separates incentives (utility) from learning behavior.

At each time step the agent’s state ℓₜ = ⟨Aₜ, Hₜ, Zₜ, Fₜ, Mₜ⟩ is updated by a (possibly stochastic) modification map Φ that takes the current state and a finite evidence object Eₜ (e.g., a batch of data, statistics, or a validation set). A modification is executed only if the agent can formally prove that it strictly increases its internal utility u, which may depend on empirical risk, resource consumption, or external constraints. The set of all states reachable under this rule is called the policy‑reachable family.

The central theoretical contribution is a sharp “policy‑level learnability boundary” (Theorem 1). The authors prove that distribution‑free PAC learnability is preserved after any sequence of self‑modifications iff the supremum of the VC dimension (or any uniform capacity measure) over the entire policy‑reachable family is finite. The sufficiency direction follows from standard uniform convergence: if a uniform capacity bound K exists, empirical risk minimization (or an AERM) yields the usual VC‑rate on the terminal predictor. The necessity direction shows that if capacities can diverge, VC lower bounds make it impossible to guarantee distribution‑free sample complexity for any learner.

To make the boundary operational, the paper proposes a “Two‑Gate” safety mechanism (Theorem 2). The first gate requires that a candidate edit produce a hypothesis whose empirical risk on an independent validation set V is lower than the current hypothesis by at least a margin τ plus a statistical slack ε_V (derived from the VC‑dimension of a pre‑specified reference family G_K