Formal Synthesis of Certifiably Robust Neural Lyapunov-Barrier Certificates

Neural Lyapunov and barrier certificates have recently been used as powerful tools for verifying the safety and stability properties of deep reinforcement learning (RL) controllers. However, existing methods offer guarantees only under fixed ideal unperturbed dynamics, limiting their reliability in real-world applications where dynamics may deviate due to uncertainties. In this work, we study the problem of synthesizing \emph{robust neural Lyapunov barrier certificates} that maintain their guarantees under perturbations in system dynamics. We formally define a robust Lyapunov barrier function and specify sufficient conditions based on Lipschitz continuity that ensure robustness against bounded perturbations. We propose practical training objectives that enforce these conditions via adversarial training, Lipschitz neighborhood bound, and global Lipschitz regularization. We validate our approach in two practically relevant environments, Inverted Pendulum and 2D Docking. The former is a widely studied benchmark, while the latter is a safety-critical task in autonomous systems. We show that our methods significantly improve both certified robustness bounds (up to $4.6$ times) and empirical success rates under strong perturbations (up to $2.4$ times) compared to the baseline. Our results demonstrate effectiveness of training robust neural certificates for safe RL under perturbations in dynamics.

💡 Research Summary

This paper addresses a critical gap in the verification of deep reinforcement‑learning (RL) controllers: existing neural Lyapunov‑Barrier certificates (also called CLBFs) guarantee safety and stability only for an ideal, unperturbed dynamics model. In real‑world settings, modeling errors, sensor noise, and environmental disturbances introduce bounded, time‑varying perturbations that can invalidate those guarantees.
The authors formally define a robust Lyapunov‑Barrier certificate for Reach‑While‑Avoid (RWA) tasks. The classic decrease condition V(x) − V(f(x,π(x))) ≥ ε is strengthened to hold for all next‑state candidates y inside an ℓp‑ball of radius δ around the nominal next state f(x,π(x)). This yields three robust CLBF conditions (initial bound, robust decrease, unsafe‑set lower bound) and a theorem proving that any trajectory satisfying them will (i) reach the goal in finite time and (ii) never enter the unsafe set, despite state‑wise perturbations.

A key theoretical insight is that if V is L‑Lipschitz continuous, then for any y in the δ‑ball we have V(y) ≤ V(f(x,π(x))) + L·δ. Consequently, the robust decrease condition is guaranteed whenever L·δ ≤ ε. This simple inequality provides a sufficient condition linking the Lipschitz constant of the certificate, the admissible perturbation size, and the required decrease margin.

To enforce these conditions during training, the paper proposes three complementary loss components:

1. Adversarial loss – explicitly searches for the worst‑case perturbed state y_adv that maximizes V within the δ‑ball and penalizes violations of V(x) − V(y_adv) ≥ ε. This yields empirical robustness.

2. Local Lipschitz loss – uses the global Lipschitz bound L (computed as the product of spectral norms of the network layers) to upper‑bound V(y) for any y in the ball, and penalizes V(x) − (V(f(x,π(x))) + L·δ) ≥ ε. This provides a certifiable robustness guarantee without explicit adversarial search.

3. Global Lipschitz regularization – directly minimizes the spectral norms of each layer, thereby reducing L itself and making the sufficient condition L·δ ≤ ε easier to satisfy.

These are combined with the standard initialization loss (enforcing V ≤ β on the initial set) and the nominal descent loss (enforcing V decrease on unperturbed transitions). Training proceeds inside a Counterexample‑Guided Inductive Synthesis (CEGIS) loop, iteratively generating counterexamples (states where the robust condition fails) and updating the network.

Experimental evaluation is performed on two benchmark RWA problems: (i) the classic Inverted Pendulum, a low‑dimensional nonlinear control task, and (ii) a 2‑D docking scenario, representing a safety‑critical robotic manipulation problem with a complex unsafe region. The authors vary the perturbation radius δ and measure (a) the certified robustness bound (the largest δ for which the robust CLBF conditions hold on the training data) and (b) the empirical success rate (percentage of simulated rollouts that reach the goal without hitting the unsafe set).

Results show that the combination of adversarial training and Lipschitz regularization yields the most substantial improvements: certified δ values increase up to 4.6× compared with the baseline neural CLBF method, and empirical success rates under strong perturbations improve up to 2.4×. Pure Lipschitz regularization also improves training stability and reduces the Lipschitz constant, but its certified bound is slightly lower than the adversarial‑augmented version.

The paper’s contributions are threefold: (1) a formal definition and theoretical guarantee for robust Lyapunov‑Barrier certificates under norm‑bounded perturbations, (2) sufficient Lipschitz‑based conditions and practical loss designs to enforce them, and (3) empirical validation demonstrating significant robustness gains on realistic control tasks.

Limitations include the conservatism of using a global Lipschitz bound (which may over‑estimate L and thus underestimate the true admissible δ), computational overhead of spectral‑norm calculations for high‑dimensional networks, and the focus on state perturbations only (input or parameter uncertainties are not addressed). Future work could explore tighter local Lipschitz estimates, scalable regularization techniques for large networks, and extensions to continuous‑time dynamics or hardware experiments.