Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Access

Asymmetric actor-critic methods are widely used in partially observable reinforcement learning, but typically assume full state observability to condition the critic during training, which is often unrealistic in practice. We introduce the informed asymmetric actor-critic framework, allowing the critic to be conditioned on arbitrary state-dependent privileged signals without requiring access to the full state. We show that any such privileged signal yields unbiased policy gradient estimates, substantially expanding the set of admissible privileged information. This raises the problem of selecting the most adequate privileged information in order to improve learning. For this purpose, we propose two novel informativeness criteria: a dependence-based test that can be applied prior to training, and a criterion based on improvements in value prediction accuracy that can be applied post-hoc. Empirical results on partially observable benchmark tasks and synthetic environments demonstrate that carefully selected privileged signals can match or outperform full-state asymmetric baselines while relying on strictly less state information.

💡 Research Summary

The paper addresses a fundamental limitation of existing asymmetric actor‑critic methods for partially observable reinforcement learning (POMDPs). Traditional approaches assume that during training the critic can access the full underlying state while the actor only receives observation‑action histories, an assumption that is often unrealistic in real‑world systems where full state information may be unavailable due to sensor limitations, privacy constraints, or computational cost.

To overcome this, the authors introduce the Informed Asymmetric Actor‑Critic (IAAC) framework, which generalizes the asymmetric setting by allowing the critic to condition on any state‑dependent privileged signal (i_t) rather than the full state. The privileged signal is drawn from a distribution (I(i_t|s_t)) and may be any auxiliary information available at training time (e.g., diagnostic sensor readings, simulator variables, forecasts). The key theoretical contribution is a set of lemmas and a theorem proving that, for any such signal, the following hold:

1. Unbiased Reward – The reward defined with the privileged signal, (R(h_t,i_t,a_t) = \mathbb{E}