Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samples

Offline reinforcement learning (RL) learns effective policies from a static target dataset. The performance of state-of-the-art offline RL algorithms notwithstanding, it relies on the size of the target dataset, and it degrades if limited samples in the target dataset are available, which is often the case in real-world applications. To address this issue, domain adaptation that leverages auxiliary samples from related source datasets (such as simulators) can be beneficial. However, establishing the optimal way to trade off the limited target dataset and the large-but-biased source dataset while ensuring provably theoretical guarantees remains an open challenge. To the best of our knowledge, this paper proposes the first framework that theoretically explores the impact of the weights assigned to each dataset on the performance of offline RL. In particular, we establish performance bounds and the existence of the optimal weight, which can be computed in closed form under simplifying assumptions. We also provide algorithmic guarantees in terms of convergence to a neighborhood of the optimum. Notably, these results depend on the quality of the source dataset and the number of samples in the target dataset. Our empirical results on the well-known Procgen and MuJoCo benchmarks substantiate the theoretical contributions in this work.

💡 Research Summary

Offline reinforcement learning (RL) promises to learn high‑performing policies from a static dataset without further interaction with the environment, a setting that is crucial for safety‑critical domains such as healthcare or autonomous driving. However, the performance of state‑of‑the‑art offline RL algorithms deteriorates sharply when the target dataset is small, which is often the case in real‑world applications. This paper tackles the problem of augmenting a limited target dataset with a large, but potentially biased, source dataset (e.g., a simulator) through a principled domain‑adaptation framework that comes with provable guarantees.

The authors formalize the problem in a tabular Markov decision process (MDP) and define the expected temporal‑difference (TD) error for a Q‑function under the target distribution ( \mathcal{D} ) and the source distribution ( \mathcal{D}’ ) as (E_{\mathcal{D}}(Q)) and (E_{\mathcal{D}’}(Q)), respectively. They propose to minimize a convex combination of the two TD‑errors:

\