Semantically Labelled Automata for Multi-Task Reinforcement Learning with LTL Instructions

We study multi-task reinforcement learning (RL), a setting in which an agent learns a single, universal policy capable of generalising to arbitrary, possibly unseen tasks. We consider tasks specified as linear temporal logic (LTL) formulae, which are commonly used in formal methods to specify properties of systems, and have recently been successfully adopted in RL. In this setting, we present a novel task embedding technique leveraging a new generation of semantic LTL-to-automata translations, originally developed for temporal synthesis. The resulting semantically labelled automata contain rich, structured information in each state that allow us to (i) compute the automaton efficiently on-the-fly, (ii) extract expressive task embeddings used to condition the policy, and (iii) naturally support full LTL. Experimental results in a variety of domains demonstrate that our approach achieves state-of-the-art performance and is able to scale to complex specifications where existing methods fail.

💡 Research Summary

The paper tackles the problem of multi‑task reinforcement learning (RL) where each task is specified by a linear temporal logic (LTL) formula. Traditional single‑task approaches construct a deterministic automaton for a given LTL specification and train a policy conditioned on the automaton’s state, but this ties the policy to a fixed task and requires retraining whenever the task changes. Recent multi‑task methods either decompose LTL specifications into subtasks or use a full automaton as a universal progress interface (UPI). The former loses contextual information, while the latter suffers from high computational cost because the entire automaton must be built and all accepting paths enumerated before execution.

The authors propose a novel task‑embedding technique based on semantic‑labelled limit‑deterministic Büchi automata (LDBA), a recent development from formal synthesis. Unlike classic LDBAs, each state in a semantic‑labelled automaton carries rich metadata: the sub‑formulae already satisfied, the set of propositions that must be observed next, and whether the state is accepting. This information makes the automaton self‑describing and enables on‑the‑fly construction—only the initial state is built at the start, and subsequent states are generated lazily as the environment produces new labels. Consequently, memory usage and runtime are dramatically reduced.

To turn these structured states into a fixed‑dimensional representation suitable for a neural policy, the authors encode the semantic label as a graph (nodes are atomic propositions and logical operators, edges encode the syntactic structure) and feed it into a Graph Neural Network (GNN). The GNN outputs an embedding vector that captures both the logical content of the current sub‑task and its progress. This embedding, concatenated with the raw MDP state, is supplied to a standard policy network that selects actions. Rewards are given when the automaton reaches an accepting state, aligning the RL objective with the probability of satisfying the LTL formula. Discount‑bias corrections from prior work on LDBA‑based RL are also applied.

Experiments span three domains: (1) ZoneEnv, a 2‑D navigation task with coloured zones; (2) a grid‑world with complex nested “until” and “release” operators; and (3) a warehouse logistics scenario involving repeated pick‑and‑place objectives. Baselines include LTL2ACTION (formula‑progression with GNN), Deep LTL (full LDBA with reach‑avoid sequence embeddings), and several recent decomposition‑based methods. The proposed approach consistently outperforms baselines in terms of learning speed, final satisfaction probability, and scalability. Notably, for specifications such as F r ∧ FG y (which requires first reaching a red zone and then perpetually visiting a yellow zone), the method achieves >95 % satisfaction while using roughly 40 % of the memory required by Deep LTL. Moreover, because the automaton is constructed incrementally, the method can handle specifications that would be infeasible to compile fully offline.

In summary, the paper makes three key contributions: (1) introducing semantic‑labelled LDBAs as a compact, expressive representation of LTL progress; (2) designing a universal progress interface that can be computed online and embedded via GNNs; and (3) demonstrating that a single policy conditioned on these embeddings can solve full‑LTL multi‑task RL problems that were previously out of reach. The work opens avenues for integrating formal methods more tightly with deep RL, especially in settings where tasks change frequently and require sophisticated temporal reasoning.