Cumulative Treatment Effect Testing under Continuous Time Reinforcement Learning

Understanding the impact of treatment effect over time is a fundamental aspect of many scientific and medical studies. In this paper, we introduce a novel approach under a continuous-time reinforcement learning framework for testing a treatment effect. Specifically, our method provides an effective test on carryover effects of treatment over time utilizing the average treatment effect (ATE). The average treatment effect is defined as difference of value functions over an infinite horizon, which accounts for cumulative treatment effects, both immediate and carryover. The proposed method outperforms existing testing procedures such as discrete time reinforcement learning strategies in multi-resolution observation settings where observation times can be irregular. Another advantage of the proposed method is that it can capture treatment effects of a shorter duration and provide greater accuracy compared to discrete-time approximations, through the use of continuous-time estimation for the value function. We establish the asymptotic normality of the proposed test statistics and apply it to OhioT1DM diabetes data to evaluate the cumulative treatment effects of bolus insulin on patients’ glucose levels.

💡 Research Summary

The paper introduces a novel framework for testing cumulative treatment effects using continuous‑time reinforcement learning (RL). Traditional methods for dynamic treatment effect analysis—especially in mobile health contexts—rely on discrete‑time models that assume regular observation intervals and often ignore long‑term carry‑over effects. To overcome these limitations, the authors model the underlying system as a continuous‑time Markov decision process (MDP) with state process (S_t) governed by a Feller‑Dynkin semigroup and infinitesimal generator (\mathcal{L}). Two deterministic policies are considered: one that always applies treatment (action 1) and one that never does (action 0). The value function for policy (a) is defined as

\