Reinforcement learning with learned gadgets to tackle hard quantum problems on real hardware

Quantum computing offers exciting opportunities for simulating complex quantum systems and optimizing large scale combinatorial problems, but its practical use is limited by device noise and constrained connectivity. Designing quantum circuits, which are fundamental to quantum algorithms, is therefore a central challenge in current quantum hardware. Existing reinforcement learning based methods for circuit design lose accuracy when restricted to hardware native gates and device level compilation. Here, we introduce gadget reinforcement learning (GRL), which combines learning with program synthesis to automatically construct composite gates that expand the action space while respecting hardware constraints. We show that this approach improves accuracy, hardware compatibility, and scalability for transverse-field Ising and quantum chemistry problems, reaching systems of up to ten qubits within realistic computational budgets. This framework demonstrates how learned, reusable circuit building blocks can guide the co-design of algorithms and hardware for quantum processors.

💡 Research Summary

The paper introduces Gadget Reinforcement Learning (GRL), a novel framework that augments traditional reinforcement‑learning (RL) approaches for quantum circuit design with program‑synthesis‑driven “gadgets” – reusable composite gates. Conventional RL methods for variational quantum algorithms (VQAs) typically operate over a fixed set of elementary gates (e.g., CNOT, RZ) and suffer from sparse rewards, large action spaces, and performance loss after transpilation to hardware‑native gate sets. GRL tackles these issues in three stages.

First, an RL agent (double‑deep Q‑network with ε‑greedy policy) builds parameterized quantum circuits (PQCs) using only the native gate set of a target device (IBM Heron: {RZ, SX, X, CZ}). The agent receives a reward based on the energy expectation of a problem Hamiltonian; a positive reward is issued when the cost falls below a dynamic threshold ζ, otherwise a negative reward is given, and the episode terminates either upon reaching ζ or after a maximum number of steps.

Second, after each training phase the top‑k performing circuits are fed into a program‑synthesis module. Circuits are expressed as typed λ‑calculus programs, and a syntax‑tree analysis extracts frequently occurring gate sequences. These sequences are evaluated using a grammar‑score that balances usage frequency (log‑likelihood) against circuit length, yielding short, high‑utility composite operations called gadgets. Gadgets may be one‑ or two‑qubit operations and are added to the RL action space, effectively expanding the dimensionality of the tensor‑based state representation without redesigning the neural network.

Third, the RL agent resumes training with the enlarged action space. Because gadgets encapsulate higher‑level structure, the agent can reach low‑energy solutions with far fewer steps, mitigating the sparse‑reward problem. The process is iterated in a curriculum fashion: the agent first solves an easy instance (e.g., a 2‑qubit transverse‑field Ising model (TFIM) with a very weak field h = 10⁻³), learns gadgets, then tackles progressively harder instances (larger h, more qubits) using the previously learned gadgets.

Empirical evaluation focuses on two benchmark families. For TFIM, GRL learns gadgets on the 2‑qubit weak‑field case and successfully applies them to 3‑qubit and 4‑qubit systems with stronger fields (h up to 1), where standard RL fails to converge. GRL achieves higher ground‑state fidelity, reduces circuit depth, and stays within the native gate set, eliminating costly transpilation. In a quantum‑chemistry test, gadgets extracted from a 2‑qubit H₂ ground‑state preparation are reused for a 3‑qubit H₂ problem, yielding a ~30 % reduction in energy error and a ~20 % shorter circuit compared to a gadget‑free baseline.

Technical contributions include: (1) a refined binary‑tensor encoding of PQCs that can be dynamically expanded when new gadgets are introduced; (2) a program‑synthesis pipeline based on syntax‑guided synthesis (SyGuS) that quantifies fragment usefulness via a grammar‑score; (3) a feedback‑driven curriculum where the threshold ζ is adaptively lowered or raised based on episode outcomes, ensuring efficient exploration.

Overall, GRL demonstrates that learning reusable, hardware‑compatible circuit building blocks can dramatically improve sample efficiency, scalability, and hardware compatibility of quantum‑circuit design. The framework paves the way for co‑design of algorithms and quantum processors, offering a path toward more practical VQAs and quantum simulations on near‑term noisy intermediate‑scale quantum (NISQ) devices. Future work may explore automatic generalization of gadgets across different qubit topologies, multi‑qubit gadget libraries, and integration with other quantum‑hardware platforms such as trapped ions or photonic processors.