Near-Optimal Second-Order Guarantees for Model-Based Adversarial Imitation Learning

We study online adversarial imitation learning (AIL), where an agent learns from offline expert demonstrations and interacts with the environment online without access to rewards. Despite strong empirical results, the benefits of online interaction and the impact of stochasticity remain poorly understood. We address these gaps by introducing a model-based AIL algorithm (MB-AIL) and establish its horizon-free, second-order sample-complexity guarantees under general function approximations for both expert data and reward-free interactions. These second-order bounds provide an instance-dependent result that can scale with the variance of returns under the relevant policies and therefore tighten as the system approaches determinism. Together with second-order, information-theoretic lower bounds on a newly constructed hard-instance family, we show that MB-AIL attains minimax-optimal sample complexity for online interaction (up to logarithmic factors) with limited expert demonstrations and matches the lower bound for expert demonstrations in terms of the dependence on horizon $H$, precision $ε$ and the policy variance $σ^2$. Experiments further validate our theoretical findings and demonstrate that a practical implementation of MB-AIL matches or surpasses the sample efficiency of existing methods.

💡 Research Summary

This paper presents a comprehensive theoretical and empirical analysis of model-based adversarial imitation learning (AIL), focusing on the online setting where an agent learns from offline expert demonstrations while simultaneously interacting with the environment without access to reward signals. The core motivation is to address the poorly understood benefits of online interaction and the impact of system stochasticity on sample complexity, despite the strong empirical performance of AIL methods.

The authors introduce a novel algorithm, Model-Based Adversarial Imitation Learning (MB-AIL). Its key design principle is the decoupling of the policy class into a reward function class and a dynamics model class. The algorithm operates in two intertwined loops: (1) An adversarial reward learning component that uses a no-regret online optimization algorithm (e.g., Online Gradient Ascent) on the offline expert data to iteratively estimate a reward function that best distinguishes the expert from the learner. (2) A model-based reinforcement learning component that uses maximum likelihood estimation (MLE) on data collected from online, reward-free interactions to build a confidence set of plausible transition models. It then performs optimistic planning within this set and the current adversarial reward to select the next policy for exploration.

The primary theoretical contribution is the establishment of horizon-free, second-order sample complexity guarantees for MB-AIL under general function approximation for both expert demonstrations and online interactions. The upper bounds scale as Õ(σ² * d_E * log|P| / ε²) for online interactions and Õ(σ_E² * log|R| / ε²) for expert demonstrations. Here, σ² and σ_E² represent the variances of the total return under the learner’s and expert’s policies, respectively, d_E is the Eluder dimension of the model class, and ε is the target suboptimality gap. These “second-order” bounds are instance-dependent and become tighter (i.e., require fewer samples) as the system becomes more deterministic (σ² → 0), precisely quantifying the effect of stochasticity.

To demonstrate the optimality of these guarantees, the paper constructs a new family of hard instances for imitation learning and derives matching information-theoretic lower bounds. The lower bounds reveal a fundamental separation: expert data is primarily essential for learning the reward function, while online interaction is inherently tied to learning the transition dynamics. A comparison shows that MB-AIL achieves minimax-optimal sample complexity for online interaction (up to logarithmic factors) given limited expert data. For expert demonstration complexity, its bound matches the lower bound except for a factor of log|R|.

Finally, the theoretical findings are validated through empirical experiments. A practical implementation of MB-AIL is tested on several benchmarks. The results demonstrate that MB-AIL matches or surpasses the sample efficiency of existing state-of-the-art imitation learning methods, such as Behavior Cloning and prior model-free AIL algorithms. The experiments particularly highlight the advantages of the model-based approach in environments with complex dynamics and under limited expert data regimes, corroborating the theoretical insights about the benefits of online exploration and the impact of variance.