Linear Temporal Logic Satisfaction in Adversarial Environments using Secure Control Barrier Certificates

Linear T emporal Logic Satisfaction in Adversarial Environments using Secure Control Barrier Certificates ? Bhaskar Ramasubramanian 1 , Luyao Niu 2 , Andrew Clark 2 , Linda Bushnell 1 , and Radha P oovendran 1 1 Department of Electrical and Computer Engineering, University of W ashington, Seattle, W A 98195, USA. {bhaskarr, lb2, rp3}@uw.edu 2 Department of Electrical and Computer Engineering, W orcester P olytechnic Institute, W orcester , MA 01609, USA. {lniu, aclark}@wpi.edu Abstract. This paper studies the satisfaction of a class of temporal prop- erties for cyber -physical systems (CPSs) over a finite-time horizon in the presence of an adversary , in an environment described by discrete-time dy- namics. The temporal logic specification is given in s a f e − L T L F , a frag- ment of linear temporal logic over traces of finite length. The interaction of the CPS with the adversary is modeled as a two-player zero-sum discrete- time dynamic stochastic game with the CPS as defender . W e formulate a dy- namic programming based approach to determine a stationary defender pol- icy that maximizes the probability of satisfaction of a s a f e − L T L F formula over a finite time-horizon under any stationary adversary policy . W e intro- duce secure control barrier certificates (S-CBCs), a generalization of barrier certificates and control barrier certificates that accounts for the presence of an adversary , and use S-CBCs to provide a lower bound on the above satisfaction probability . When the dynamics of the evolution of the system state has a specific underlying structure, we present a way to determine an S-CBC as a polynomial in the state variables using sum-of-squares opti- mization. An illustrative example demonstrates our approach. Keywords: Linear temporal logic · sa f e − L T L F · Dynamic programming · Secure control barrier certificate · Sum-of-squares optimization. 1 Introduction Cyber -physical systems (CPSs) use computing devices and algorithms to inform the working of a physical system [8]. These systems are ubiquitous, and vary in size and scale from energy systems to medical devices. The wide-spread influence ? This work was supported by the U .S . Army Research Office, the National Science F oun- dation, and the Office of Naval Researc h via Grants W911NF-16-1-0485, CNS-1656981, and N00014-17-S-B001 respectively . of CPSs such as power systems and automobiles makes their safe operation criti- cal. Although distributed algorithms and systems allow for more efficient sharing of information among parts of the system and across geographies , they also make the CPS vulnerable to attacks by an adversary who might gain access to the dis- tributed system via multiple entry points . Attacks on distributed CPSs ha ve been reported across multiple application domains [20], [43], [44], [46]. In these cases, the damage to the CPS was caused by the actions of a stealthy , intelligent ad- versary . Thus, methods designed to only account for modeling and sensing errors may not meet performance requirements in adversarial scenarios. Therefore, it is important to develop ways to specify and verify properties that a CPS must sat- isfy that will allow us to provide guarantees on the operation of the system while accounting for the presence of an adversary . In order to verify the behavior of a CPS against a rich set of temporal speci- fications, techniques from formal methods can be used [9]. Properties like safety , stability , and priority can be expressed as formulas in linear temporal logic (L TL) [19]. These properties can then be verified using off-the-shelf model solvers [15], [28] that take these formulas as inputs. If the state space and the actions avail- able to the agents are both finite and discrete, then the environment can be repre- sented as a Markov decision process (MDP) [38] or a stochastic game [11]. These representations have also been used as abstractions of continuous-state continu- ous action dynamical system models [10], [32]. However , a significant shortcoming is that the computational complexity of abstracting the underlying system grows exponentially with the resolution of discretization desired [14], [21]. The method of barrier certificates (or barrier functions), which are functions of the states of the system was introduced in [36]. Barrier functions provide a cer- tificate that all trajectories of a system starting from a given initial set will not enter an unsafe region. The use of barrier functions does not require explicit com- putation of sets of reac hable states, whic h is known to be undecidable for general dynamical systems [29], and moreover , it allows for the analysis of general nonlin- ear and stochastic dynamical systems. The authors of [36] further showed that if the states and inputs to the system have a particular structure, computationally efficient methods can be used to construct a barrier certificate. Barrier certificates were used to determine probabilistic bounds on the satis- faction of an L TL formula by a discrete-time stochastic system in [22]. A more re- cent work by the same authors [23] used control barrier certificates to synthesize a policy in order to maximize the probability of satisfaction of an L TL formula. Prior work that uses barrier certificates to study temporal logic satisfaction assumes a single agent, and does not study the case when the CPS is operating in an adversarial environment. T o the best of our knowledge, this paper is the first to use barrier certificates to study temporal logic satisfaction for CPSs in adversarial environments . W e introduce secure barrier certificates (S-CBCs), and use it to determine probabilistic bounds on the satisfaction of an L TL formula under any adversary policy . Further , definitions of barrier certificates and control barrier certificates in prior work can be recovered as special cases of S-CBCs. 1.1 Contributions In this paper , we consider the setting when there is an adversary whose aim is to ensure that the L TL formula is not satisfied by the CPS (defender). The temporal logic specification is given in s a f e − L T L F , a fragment of L TL over traces of finite length. W e make the following contributions: – W e model the interaction between the CPS and adversary as a two-player dynamic stochastic game with the CPS as defender . The two players take their actions simultaneously , and these jointly influence the system dynamics. – W e present a dynamic programming based approach to determine a station- ary defender policy to maximize the probability of satisfaction of an L TL for- mula over a finite time-horizon under any stationary adversary policy . – In order to determine a lower bound on the above satisfaction probability , we define a new entity called secure control barrier certificates (S-CBCs) . S-CBCs generalize barrier certificates and control barrier certificates to account for the presence of an adversary . – When the evolution of the state of the dynamic game can be expressed as poly- nomial functions of the states and inputs, we use sum-of-squares optimization to compute an S-CBC as a polynomial function of the states. – W e present an illustrative example demonstrating our approach. 1.2 Outline of P aper W e summarize related work on control barrier certificates and temporal logic sat- isfaction in Section 2. Section 3 gives an overview of temporal logic and game- theoretic concepts that will be used to derive our results. The problem that is the focus of this paper is formulated in Section 4. Our solution approach is presented in Section 5, where we define a dynamic programming operator to synthesize a policy for the defender in order to maximize the probability of satisfaction of the L TL formula under any adversary policy . W e define a notion of secure control barrier certificates to derive a lower bound on the satisfaction probability , and are able to explicitly compute an S-CBC under certain assumptions . Section 6 presents an illustrative example, and we conclude the paper in Section 7. 2 Related W ork The method of barrier functions was introduced in [36] to certify that all trajec- tories of a continuous-time system starting from a given initial set do not enter an unsafe region. Control barrier functions (CBFs) were used to provide guaran- tees on the safety of continuous-time nonlinear systems with affine inputs for an adaptive cruise control application in [6]. The notion of input-to-state CBFs that ensured the safety of nonlinear systems under arbitrary input disturbances was introduced in [24], and safety was characterized in terms of the invariance of a set whose computation depended on the magnitude of the disturbance. The authors of [45] relaxed the supermartingale condition that a barrier certificate had to sat- isfy in [36] in order to provide finite-time guarantees on the safety of a system. The verification and control of a finite-time safety property for continuous-time stochastic systems using barrier functions was recently presented in [41]. Barrier certificates were used to verify L TL formulas for a deterministic, continuous-time nonlinear dynamical system in [49]. Time-varying CBFs were used to accomplish tasks specified in signal temporal logic in [30]. A survey of the use of CBFs to design safety-critical controllers is presented in [5]. The use of barrier certificates or CBFs in these works were all for continuous time dynamical systems and did not consider the effect of the actions of an adversarial player . Barrier certificates in the discrete-time setting were used to analyze the reach- able belief space of a partially observable Markov decision process (POMDP) with applications to verifying the safety of POMDPs in [2], and for privacy verification in POMDPs in [3]. The use of barrier certificates for the verification and synthesis of control policies for discrete-time stochastic systems to satisfy an L TL formula over a finite time horizon was presented in [22] and [23]. These papers also as- sumed a single agent, and did not account for the presence of an adversary . The authors of [33] used barrier functions to solve a reference tracking prob- lem for a continuous-time linear system subject to possible false data injection attacks by an adversary , with additional constraints on the safety and reachabil- ity of the system. Probabilistic reachability over a finite time horizon for discrete- time stochastic hybrid systems was presented in [1]. This was extended to a dy- namic stochastic game setting when there were two competing agents in [18], and to the problem of ensuring the safety of a system that was robust to errors in the probability distribution of a disturbance input in [50]. These papers did not assume that a temporal specification had to be additionally satisfied. Determining a policy for an agent in order to maximize the probability of sat- isfying an L TL formula in an environment specified by an MDP was presented in [19]. This setup w as extended to the case when there were two agents- a defender and an adversary- who had competing objectives to ensure the satisfaction of the L TL formula in an environment specified as a stochastic game in [32]. These pa- pers assume that the states of the system are completely observable, which might not be true in every situation. The satisfaction of an L TL formula in partially observable environments represented as POMDPs was studied in [42] and the ex- tension to partially observable stochastic games with two competing agents , each with its own observation of the state of the system, was formulated in [39]. 3 Preliminaries In this section, we give a brief introduction to linear temporal logic and discrete- time dynamic stochastic games. Wherever appropriate, we consider a probability space ( Ω , F , P ). W e write ( X , B ( X )) to denote the measurable space X equipped with the Borel σ − algebra, and R ≥ 0 to denote the set of non-negative real numbers. 3.1 Linear T emporal Logic T emporal logic frameworks enable the representation and reasoning about tem- poral information on propositional statements . Linear temporal logic (L TL) is one such framework, where the progress of time is ‘linear’. An L TL formula [9] is defined over a set of atomic propositions A P , and can be written as: φ : = T | σ |¬ φ | φ ∧ φ | X φ | φ U φ , where σ ∈ A P , and X and U are temporal operators denoting the next and until operations. The semantics of L TL are defined over (infinite) words in 2 A P . The syntax of linear temporal logic over finite traces, denoted L T L F [17], is the same as that of L TL. The semantics of L T L F is expressed in terms of finite- length words in 2 A P . W e denote a word in L T L F by η , write | η | to denote the length of η , and η i , 0 < i < | η | , to denote the proposition at the i t h position of η . W e write ( η , i ) | = φ when the L T L F formula φ is true at the i t h position of η . Definition 1 ( L T L F Semantics). The semantics of L T L F can be recursively de- fined in the following way: 1. ( η , i ) | = T ; 2. ( η , i ) | = σ iff σ ∈ η i ; 3. ( η , i ) | = ¬ φ iff ( η , i ) 6| = φ ; 4. ( η , i ) | = φ 1 ∧ φ 2 iff ( η , i ) | = φ 1 and ( η , i ) | = φ 2 ; 5. ( η , i ) | = X φ iff i < | η | − 1 and ( η , i + 1) | = φ ; 6. ( η , i ) | = φ 1 U φ 2 iff ∃ j ∈ [ i , | η | ] such that ( η , j ) | = φ 2 and for all k ∈ [ i , j ) , ( η , k ) | = φ 1 . Finally , we write η | = φ if and only if ( η , 0) | = φ . Moreover , the logic admits derived formulas of the form: i) φ 1 ∨ φ 2 : = ¬ ( ¬ φ 1 ∧ ¬ φ 2 ); ii) φ 1 ⇒ φ 2 : = ¬ φ 1 ∨ φ 2 ; iii) F φ : = T U φ (eventually); iv) G φ : = ¬ F ¬ φ (alwa ys). The set L ( φ ) comprises the language of finite-length words associated with the L T L F formula φ . In this paper , we focus on a subset of LT L F called s a f e − L T L F [40], that explicitly considers only safety properties [26]. Definition 2 ( s a f e − L T L F F ormula). An L T L F formula is a s a f e − L T L F for - mula if it can be written in positive normal form (PNF) 3 , using the temporal oper - ators X (next) and G (always). Next, we define an entity that will serve as an equivalent representation of an L T L F formula, and will allow us to c heck if the L T L F formula is satisfied or not. Definition 3 (Deterministic Finite Automaton). A deterministic finite au- tomaton (DF A) is a quintuple A = ( Q , Σ , δ , q 0 , F ) where Q is a nonempty finite set of states , Σ is a finite alphabet, δ : Q × Σ → Q is a transition function, q 0 ∈ Q is the initial state , and F ⊆ Q is a set of accepting states . 3 In PNF , negations occur only adjacent to atomic propositions. Definition 4 (Accepting Runs). A run of A of length n is a finite sequence of ( n + 1) states q 0 σ 0 − → q 1 σ 1 − → . . . σ n − 1 − − − → q n such that q i ∈ δ ( q i − 1 , σ i − 1 ) for all i ∈ [1 , n ] and for some σ 0 , . . . , σ n − 1 ∈ Σ . The run is accepting if q n ∈ F . W e write L ( A ) to denote the set of all words accepted by A . Every L T L F formula φ over A P can be represented by a DF A A φ with Σ = 2 A P that accepts all and only those runs that satisfy φ , that is, L ( φ ) = L ( A φ ) [16]. The DF A A φ can be constructed by using a tool like Rabinizer4 [25]. 3.2 Discrete-time Dynamic Stochastic Games W e model the interaction between the CPS (defender) and adversary as a two- player dynamic stochastic game that evolves according to some known (discrete- time) dynamics [7]. The evolution of the state of the game at each time step is affected by the actions of both players . Definition 5 (Discrete-time Dynamic Stochastic Game). A discrete-time dy- namic stochastic game (DDSG) is a tuple G = ( X , W , U d , U a , f , N , A P , L ) , where X ⊆ R n and W are Borel-measurable spaces representing the state-space and un- certainty space of the system, U d ⊆ R d and U a ⊆ R a are compact Borel spaces that denote the action sets of the defender and adversary , f : X × U d × U a × W → X is a Borel-measurable transition function characterizing the evolution of the system, N = { 0 , 1 , . . . , N − 1 } is an index-set denoting the stage of the game , A P is a set of atomic propositions , and L : X → 2 A P is a labeling function that maps states to a subset of atomic propositions that are satisfied in that state . The evolution of the state of the system is given by: x ( k + 1) = f ( x ( k ) , u d ( k ) , u a ( k ) , w ( k )); x (0) = x 0 ∈ X ; k ∈ N , (1) where { w ( k ) } is a sequence of independent and identically distributed (i.i.d.) ran- dom variables with zero mean and bounded covariance. In this paper , we focus on the Stackelberg setting with the defender as leader and adversary as follower . The leader selects its inputs anticipating the worst- case response by the adversary . W e assume that the adversary can choose its action based on the action of the defender [18], and further , restrict our focus to stationary strategies for the two players. Due to the asymmetry in information available to the players, equilibrium strategies for the case when the game is zero-sum can be chosen to be deterministic strategies [13]. Definition 6 (Defender Strategy). A stationary strategy for the defender is a sequence µ ( d ) : = { µ ( d ) k } k ∈ N of Borel-measurable maps µ ( d ) k : X → U d . Definition 7 (Adversary Strategy). A stationary strategy for the adversary is a sequence µ ( a ) : = { µ ( a ) k } k ∈ N of Borel-measurable maps µ ( a ) k : X × U d → U a . 4 Problem F ormulation F or a DDSG G , recall that the labeling function L indicates which atomic propo- sitions are true in each state. Assumption 1 W e restrict our attention to labeling functions of the form L : X → A P . Then, if A P = ( a 1 , . . . , a p ) , A P and L will partition the state space as X : = ∪ p i = 1 X i , where X i : = L − 1 ( a i ) . W e further assume that X i 6= ; for all i . Remark 1. Through the remainder of the paper , we interchangeably use x k or x ( k ) to denote the state at time k . Given a sequence of states x N : = ( x 0 , x 1 , . . . , x N − 1 ), using Assumption 1, if η k = L ( x k ) for all k ∈ N , then we can write L ( x N ) = ( η 0 , η 1 , . . . , η N − 1 ). Definition 8 (L TL Satisfaction by DDSG). F or a DDSG G and a s a f e − L T L F formula φ , we write P x 0 µ ( d ) , µ ( a ) { L ( x N ) | = φ } to denote the probability that the evolu- tion of the DDSG starting from x (0) = x 0 under player policies µ ( d ) and µ ( a ) satisfies φ over the time horizon N = { 0 , 1 , . . . , N − 1 } . W e are now ready to formally state the problem that this paper seeks to solve . Problem 1. Given a discrete-time dynamic game G = ( X , W , U d , U a , f , N , A P , L ) that evolves according to the dynamics in Equation (1) and a sa f e − L T L F for - mula φ , determine a policy f or the defender , µ ( d ) , that maximizes the probability of satisfying φ over the time horizon N = { 0 , 1 , . . . , N − 1 } under any adversary policy µ ( a ) for all x 0 ∈ L − 1 ( a j ) for some a j ∈ A P . That is , compute: sup µ ( d ) inf µ ( a ) P x 0 µ ( d ) , µ ( a ) { L ( x N ) | = φ } (2) 5 Solution Approach In this section, we present a dynamic programming approach to determine a so- lution to Problem 1. Our analysis is motivated by the treatment in [18] and [50]. W e then introduce the notion of secure control barrier certificates (S-CBCs), and use these to provide a lower bound on the probability of satisfaction of the s a f e − L T L F formula φ for a defender policy under any adversary policy in terms of the accepting runs of length less than or equal to the length of the time-horizon of interest of a DFA associated with φ . F or systems whose evolution of states can be written as a polynomial function of states and inputs, we present a sum-of- squares optimization approach in order to compute an S-CBC . S-CBCs generalize barrier certificates [22] and control barrier certificates [23] to account for the presence of an adversary . A difference between the treatment in this paper and that of [22], [23] is that we define S-CBCs for stochastic dynamic games, while the latter papers focus on stoc hastic systems with a single agent. 5.1 Dynamic Programming for s a f e − L T L F Satisfaction W e introduce a dynamic programming (DP) operator that will allow us to recur- sively solve a Bellman equation related to Equation (2) backward in time. First, observe that we can write the satisfaction probability in Definition 8 as: P x 0 µ ( d ) , µ ( a ) { L ( x N ) | = φ } = E µ ( d ) , µ ( a ) { Y k ∈ N 1 ( L ( x k ) | = φ ) | x (0) = x 0 } , (3) where E µ ( d ) , µ ( a ) is the expectation operator under the probability measure P µ ( d ) , µ ( a ) induced by agent policies µ ( d ) and µ ( a ) . 1 ( · ) is the indicator function, which takes value 1 if its argument is true, and 0 otherwise . Assume that V : X → [0 , 1] is a Borel-measurable function. A DP operator T can then be characterized in the following wa y: V ( x N − 1 ) = 1 ( L ( x N − 1 ) | = φ ) (4) ( T V )( x k ) : = sup u d inf u a 1 ( L ( x k ) | = φ ) Z X V ( f ( x k , u d , u a , w )) d x k + 1 , (5) where d x k + 1 ≡ ( d x k + 1 | x k , u d , u a ) is a probability measure on the Borel space ( X , B ( X )). The following results adapts Theorem 1 of [18] to the case of temporal logic formula satisfaction over a finite time-horizon. Theorem 1. Assume that the DDSG G has to satisfy a s a f e − LT L F formula φ over horizon N . Let the DP operator T be defined as in Equation (5). Additionally , if d x k ≡ ( d x k + 1 | x k , u d , u a ) is continuous , then, sup µ ( d ) inf µ ( a ) P x 0 µ ( d ) , µ ( a ) { L ( x N ) | = φ } = ( T N V )( x 0 ) , (6) where T N : = T ◦ T ◦ · · · ◦ T ( N times) is the repeated composition of the operator T . Proof . Consider a particular pair of stationary agent policies µ ( d ) and µ ( a ) . F or these policies, define measurable functions V µ ( d ) , µ ( a ) k : X → [0 , 1], k = 0 , 1 , . . . , N − 1: V µ ( d ) , µ ( a ) N − 1 ( x N − 1 ) : = 1 ( L ( x N − 1 ) | = φ ) (7) V µ ( d ) , µ ( a ) k ( x k ) : = E µ ( d ) , µ ( a ) { N − 1 Y i = k 1 ( L ( x i ) | = φ ) | x ( k ) = x k } , k = 0 , 1 , . . . , N − 2 (8) Therefore, we ha ve P x 0 µ ( d ) , µ ( a ) { L ( x N ) | = φ } = V µ ( d ) , µ ( a ) 0 ( x 0 ). Now , consider strategies of the agents at a stage k . Define the operator T µ ( d ) k , µ ( a ) k : ( T µ ( d ) k , µ ( a ) k V )( x k ) : = 1 ( L ( x k ) | = φ ) Z X V ( f ( x k , u d , u a , w )) d x k + 1 (9) Expanding Equation (8) using the definition of the expectation operator will allow us to write V µ ( d ) , µ ( a ) k ( x ) = ( T µ ( d ) k + 1 , µ ( a ) k + 1 V )( x ). The result follows by an induction argument which uses the fact that T µ ( d ) k , µ ( a ) k is a monotonic operator . W e refer to [18] for details. Further , this procedure also guarantees the existence of a defender policy that will maximize the probability of satisfaction of φ under any adversary policy . u t 5.2 Secure Control Barrier Certificates Definition 9. A continuous function B : X → R ≥ 0 is a secure control barrier cer - tificate (S-CBC) for the DDSG G if for any state x ∈ X and some constant c ≥ 0 , inf u d sup u a E w [ B ( f ( x , u d , u a , w ) | x ] ≤ B ( x ) + c . (10) Intuitively , for some defender action u d , the increase in the value of an S-CBC is bounded from above along trajectories of G under any adversary action u a . Remark 2. S-CBCs generalize control barrier certificates and barrier certificates seen in prior work. If f ( x , u d , u a 1 , w ) ∼ f ( x , u d , u a 2 , w ) for every u a 1 , u a 2 ∈ U a , then we recover the definition of a control barrier certificate [23]. The definition of a barrier certificate [22], [36] is got by additionally requiring that f ( x , u d 1 , u a 1 , w ) ∼ f ( x , u d 2 , u a 2 , w ) for every u d 1 , u d 2 ∈ U d and u a 1 , u a 2 ∈ U a . Here ∼ denotes stochas- tic equivalence of the respective stochastic processes [35]. In the latter case, when c = 0, the function B is a super -martingale . F or this case, along with some addi- tional assumptions on the system dynamics, asymptotic guarantees on the satis- faction of properties over the infinite time-horizon can be established [36]. Remark 3. Although our definition of S-CBCs in Definition 9 bears resemblance to the notion of a worst-case barrier certificate introduced in [36], there are some distinctions. While the entity in [36] considers a dynamical system with a single disturbance input, our setting considers three terms that influence the evolution of the state of the system: we want to find a defender input that will allow the barrier function to satisfy a certain property under any adversary input and dis- turbance. A second point of difference is that while [36] focuses on asymptotic analysis, we consider properties over a finite time horizon. W e limit our attention to stationary strategies for both players. Studying the effects of other strategies is left as future work. The following preliminary result will be used subsequently to determine a bound on the probability of reaching a subset of states under particular agent policies over a finite time-horizon. Lemma 1. Consider a DDSG G and let B : X → R ≥ 0 be an S-CBC as in Definition 9 with constant c ≥ 0 . T hen, for any λ > 0 and initial state x 0 ∈ X , for a stationary defender policy , µ ( d ) : X → U d , the following holds under any stationary adversary policy µ ( a ) : X × U d → U a : inf µ ( d ) sup µ ( a ) P x 0 µ ( d ) , µ ( a ) [ sup 0 ≤ k < N B ( x ( k )) ≥ λ ] ≤ B ( x 0 ) + c N λ (11) Proof . The proof follows from the result of Chapter III, Theorem 3 and Corollary 2-1 in [27], Definition 9, and the fact that the agents adopt stationary policies . u t Definition 10 ( s − Reachability). F or the DDSG G with dynamics in Equation (1), let s ∈ [0 , 1] and X 0 ⊂ X be the set of possible initial states and X 1 ⊂ X be disjoint from X 0 . T hen, given x 0 ∈ X 0 , G is s − reachable with respect to X 1 , if sup k ∈ N P x 0 [ x k ∈ X 1 ] ≤ s . That is , the probability of reaching a state in X 1 starting from x 0 ∈ X 0 in the time horizon [0 , N ] is upper bounded by s . Theorem 2. With X 0 and X 1 known, and X 0 ∩ X 1 = ; , assume there exists an S-CBC B : X → R ≥ 0 , stationary policies , µ ( d ) : X → U d and µ ( a ) : X × U d → U a , and constant c ≥ 0 . Additionally , if there is a constant δ ∈ [0 , 1] such that: 1. B ( x ) ≤ δ for all x ∈ X 0 , 2. B ( x ) > 1 for all x ∈ X 1 , then the DDSG G starting from x 0 ∈ X 0 is ( δ + c N ) − reachable with respect to X 1 . Proof . Observe that X 1 ⊆ { x ∈ X : B ( x ) ≥ 1 } . Therefore, starting from x 0 , and follow- ing the respective agent policies, P x 0 µ ( d ) , µ ( a ) [ ∃ k ∈ N : x ( k ) ∈ X 1 ] ≤ P x 0 µ ( d ) , µ ( a ) [ B ( x ( k )) ≥ 1]. Since this should be true for arbitrary k , we have: sup k ∈ N P x 0 [ x k ∈ X 1 ] ≤ P x 0 µ ( d ) , µ ( a ) { sup k ∈ N B ( x ( k )) ≥ 1 } ≤ inf µ ( d ) sup µ ( a ) P x 0 µ ( d ) , µ ( a ) { sup k ∈ N B ( x ( k )) ≥ 1 } ≤ B ( x 0 ) + c N ≤ δ + c N The second line of the above system of inequalities follows by setting λ = 1 in Lemma 1, and the fact that B ( x ) ≤ δ for all x ∈ X 0 . u t 5.3 Automaton-Based V erification In order to verify that { L ( x N ) | = φ } under agent policies µ ( d ) and µ ( a ) , we need to establish that ( η 0 , η 1 , . . . , η N − 1 ) ⊆ L ( A φ ). T o do this, we first construct a DFA A ¬ φ , that accepts all and only those words over A P that do not satisfy the s a f e − L T L F formula φ . W e have the following result: Lemma 2. [9] F or L ( x N ) = ( η 0 , η 1 , . . . , η N − 1 ) and a DF A A φ , the following is true: ( η 0 , η 1 , . . . , η N − 1 ) ⊆ L ( A φ ) ⇔ ( η 0 , η 1 , . . . , η N − 1 ) ∩ L ( A ¬ φ ) = ; The construction of A ¬ φ can also be carried out in Rabinizer4 [25]. The ac- cepting runs of A ¬ φ of length less than or equal to N can be computed using a depth-first search algorithm [47]. F or the purposes of this section, it is important to understand that the accepting runs of A ¬ φ of length less than or equal to N will give a bound on the probability that a particular pair of agent policies ( µ ( d ) , µ ( a ) ) will not satisfy φ over the time horizon N . Using Definition 4 and following the treatment of [22] and [23] define the following terms (the reader is also referred to these works for an example that offers a detailed treatment of the procedure): R N ( A ¬ φ ) : = { q = ( q 0 , . . . , q n ) ∈ L ( A ¬ φ ) : n ≤ N , q i 6= q i + 1 ∀ i < n } (12) R a N ( A ¬ φ ) : = { q = ( q 0 , . . . , q n ) ∈ R N ( A ¬ φ ) : a ∈ A P and q 0 a − → q 1 } (13) P a ( q ) : = ( { ( q i , q i + 1 , q i + 2 , T ( q , q i + 1 )) : 0 ≤ i ≤ n − 2 } q ∈ R a N ( A ¬ φ ) , | q | > 2 ; o t h e r w i s e (14) T ( q , q i + 1 ) : = ( N + 2 − | q | ∃ a ∈ A P : q i + 1 a − → q i + 1 1 o t h e r w i s e (15) Intuitively , R N ( A ¬ φ ) is the set of accepting runs in A ¬ φ of length not greater than N , and without counting any self-loops in the states of the DF A. The set R a N ( A ¬ φ ) is the set of runs in R N ( A ¬ φ ) with the first state transition labeled by a ∈ A P . F or an element of R a N ( A ¬ φ ), P a ( q ) defines the set of paths of length 3 augmented with a ‘loop-bound’. The ‘loop-bound’ T ( q , q i + 1 ) is an indicator of the number of ‘self-loops’ the run in the DF A can make at state q i + 1 while still keeping its length less than or equal to N . W e assume that T ( q , q i + 1 ) = 1 when the run cannot make a self-loop at q i + 1 . 5.4 Satisfaction probability using S-CBCs and A ¬ φ In this section, we show that an accepting run of A ¬ φ of length less than or equal to N gives a lower bound on the probability that a particular pair of agent policies will not satisfy the s a f e − L T L F formula φ . W e use this in conjunction with the S-CBC to derive an upper bound on the probability that φ will be satisfied for a particular choice of defender policy under any adversary policy . Specifically , we use Theorem 2 over each accepting run of A ¬ φ of length less than or equal to N to give a bound on the overall satisfaction probability . Theorem 3. Assume that the DDSG G has to satisfy a s a f e − LT L F formula φ over horizon N . Let A ¬ φ be the DFA corresponding to the negation of φ , and for this DF A, assume that the quantities in Equations (12)-(14) have been computed. Then, for some a j ∈ A P and all x 0 ∈ L − 1 ( a j ) the maximum value of the probabil- ity of satisfaction of φ for a defender policy µ ( d ) under any adversary policy µ ( a ) satisfies the following inequality: sup µ ( d ) inf µ ( a ) P x 0 µ ( d ) , µ ( a ) { L ( x N ) | = φ } ≥ 1 − X q ∈ R a j N ( A ¬ φ ) Y ρ ∈ P a j ( q ) ( δ ρ + c ρ T ) , where ρ = ( q , q 0 , q 00 , T ) ∈ P a j ( q ) is the set of paths of length 3 with loop bound T for a j ∈ A P in an accepting run of length N in A ¬ φ . Proof . F or a j ∈ A P , consider q ∈ R a j N ( A ¬ φ ) (Equation (13)) and the set P a j ( q ) (Equations (14) and (15)). Consider an element ρ = ( q , q 0 , q 00 , T ) ∈ P a j ( q ). From Theorem 2, for some stationary defender policy µ ( d ) , the probability that a tra- jectory of G starting from x 0 ∈ L − 1 ( σ : q σ − → q 0 ) and reaching x 1 ∈ L − 1 ( σ : q 0 σ − → q 00 ) under stationary adversary policy µ ( a ) over the time horizon T is at most δ ρ + c ρ T . Therefore, the probability of an accepting run in A ¬ φ of length at most N starting from x 0 ∈ L − 1 ( a j ) is upper bounded by: inf µ ( d ) sup µ ( a ) P x 0 µ ( d ) , µ ( a ) { L ( x N ) | = ¬ φ } ≤ X q ∈ R a j N ( A ¬ φ ) Y ρ ∈ P a j ( q ) ( δ ρ + c ρ T ) Now consider Equation (2) of Problem 1. W e have the following set of equiva- lences and inequalities: sup µ ( d ) inf µ ( a ) P x 0 µ ( d ) , µ ( a ) { L ( x N ) | = φ } = sup µ ( d ) ( − sup µ ( a ) ( − P x 0 µ ( d ) , µ ( a ) { L ( x N ) | = φ } )) = − inf µ ( d ) sup µ ( a ) ( − P x 0 µ ( d ) , µ ( a ) { L ( x N ) | = φ } ) = − inf µ ( d ) sup µ ( a ) ( − 1 + P x 0 µ ( d ) , µ ( a ) { L ( x N ) | = ¬ φ } ) ≥ 1 − inf µ ( d ) sup µ ( a ) P x 0 µ ( d ) , µ ( a ) { L ( x N ) | = ¬ φ } ≥ 1 − X q ∈ R a j N ( A ¬ φ ) Y ρ ∈ P a j ( q ) ( δ ρ + c ρ T ) u t Theorem 3 generalizes Theorem 5.2 of [23] to provide a lower bound for a station- ary defender policy that maximizes the probability that the s a f e − L T L F formula is satisfied by the DDSG G over the time horizon N , starting from x 0 ∈ L − 1 ( a j ) for some a j ∈ A P for any stationary adversary policy . 5.5 Computing an S-CBC The use of barrier functions will circumvent the need to explicitly compute sets of reachable states, whic h is known to be undecidable for general dynamical systems [29]. However , computationally efficient methods can be used to construct a bar- rier certificate if the system dynamics can be expressed as a polynomial [36]. This will allow for determining bounds on the probability of satisfaction of the L TL formula without discretizing the state space. In contrast, if the underlying state space is continuous , computing the satisfaction probability and the corresponding agent policy using dynamic programming will necessitate a discretization of the state space in order to approximate the integral in Equation (5). W e propose a sum-of-squares (SOS) optimization [34] based approach that will allow us to compute an S-CBC if the evolution of the state of the DDSG has a specific structure. The key insight is that if a function can be written as a sum of squares of different polynomials, then it is non-negative . Assumption 2 The sets X , U d , U a in the DDSG G are continuous , and f ( x , u d , u a , w ) in Equation (1) can be written as a polynomial in x , u d , u a for any w . Further , the sets X i = L − 1 ( a i ) in Assumption 1 can be represented by polynomial inequalities . Proposition 1. Under the conditions of Assumption 2, suppose that sets X 0 : = { x ∈ X : g 0 ( x ) ≥ 0 } , X 1 : = { x ∈ X : g 1 ( x ) ≥ 0 } , and X : = { x ∈ X : g ( x ) ≥ 0 } , where the inequalities are element-wise . Assume that there is an SOS polynomial B ( x ) , constants δ ∈ [0 , 1] and c , SOS (vector) polynomials s 0 ( x ) , s 1 ( x ) , and s ( x ) , and pol y- nomials s d u i ( x ) corresponding to the i t h entry in u d , such that: − B ( x ) − s Ö 0 ( x ) g 0 ( x ) + δ (16) B ( x ) − s Ö 1 ( x ) g 1 ( x ) − 1 (17) ∀ u a ∈ U a : − E w [ B ( f ( x , u d , u a , w ) | x ] + B ( x ) − X i ( u d i − s d u i ( x )) − s Ö ( x ) g ( x ) + c (18) are all SOS polynomials . Then, B ( x ) satisfies the conditions of T heorem 2, and u d i = s d u i ( x ) is the corresponding defender policy . Proof . The proof of this result follows in a manner similar to Lemma 7 in [49] and Lemma 5.6 in [23], and we do not present it here. u t The authors of [23] discuss an alternative approach in the case when the input set has finite cardinality . A similar treatment is beyond the scope of the present paper , and will be an interesting future direction of research. 6 Example W e present an example demonstrating our solution approach to Problem 1. Example 1. Let the dynamics of the DDSG G with X = W = R 2 , U d is a compact subset of R , U a = [ − 1 , 1], and w 1 ( k ) , w 2 ( k ) ∼ U n i f [ − 1 , 1] (and i.i.d.) be given by: x 1 ( k + 1) = − 0 . 5 x 1 ( k ) x 2 ( k ) + w 1 ( k ) (19) x 2 ( k + 1) = x 1 ( k ) x 2 ( k ) + 0 . 1 x 2 2 ( k ) + u d ( k ) + 0 . 6 u a ( k ) + w 2 ( k ) (20) Let A P = { a 0 , a 1 , a 2 , a 3 , a 4 } , and sets X 0 , X 1 , X 2 , X 3 , X 4 such that for x ∈ X i , L ( x ) = a i . The sets X i are defined by: X 0 : = { ( x 1 , x 2 ) : x 2 1 + x 2 2 ≤ 0 . 9 } , X 1 : = { ( x 1 , x 2 ) : (2 ≤ x 1 ≤ 6) ∧ ( − 2 ≤ x 2 ≤ 2) } , X 2 : = { ( x 1 , x 2 ) : x 2 1 + ( x 2 − 10) 2 ≤ 4 } , X 3 : = { ( x 1 , x 2 ) : ( − 10 ≤ x 1 ≤ − 3) ∧ ( − 4 ≤ x 2 ≤ − 2) } , X 4 : = X \ [ i X i . The aim for an agent is to determine a sequence of inputs { u d } such that start- ing from X 0 , for any sequence of adversary inputs { u a } , it avoids obstacles in its environment, defined by the sets X 1 , X 2 , and X 3 for 10 units of time. The corre- sponding sa f e − L T L F formula is φ = [ a 0 ∧ G ¬ ( a 1 ∨ a 2 ∨ a 3 )]. The DF A that accepts ¬ φ is shown in Figure 1. Suppose we are interested in determining a bound on the probability of φ being satisfied for a time-horizon of length 10. Using Equations (12) - (15), we have P a 0 ( q 0 , q 1 , q 2 ) = { ( q 0 , q 1 , q 2 , 9) } , and P a j = ; for j = 1 , 2 , 3 , 4. W e use a sum-of-squares optimization toolbox, SOST OOLS [37] along with SDPT3 [48], a semidefinite program solver . The barrier function B ( x ) = B ( x 1 , x 2 ) was assumed to be a polynomial of degree-two. F or the case c = 0, we determine the smallest value of δ that will satisfy the conditions in Proposition 1 to compute an S-CBC . The output of the program was an S-CBC given by B ( x ) = 0 . 1915 x 2 1 + 0 . 1868 x 1 x 2 − 0 . 144 x 1 + 0 . 1201 x 2 2 + 0 . 1239 x 2 + 0 . 16 The environment and the obstacles denoted by the sets X 1 , X 2 , X 3 and the contours of the S-CBC is shown in Figure 2. W e observe that B ( x ) is less than 1 in Fig . 1: The DF A that accepts ¬ φ for the s a f e − L T L F formula φ = [ a 0 ∧ G ¬ ( a 1 ∨ a 2 ∨ a 3 )] and A P = { a 0 , a 1 , a 2 , a 3 , a 4 } . Fig . 2: The regions X 0 , X 1 , X 2 , X 3 , X 4 along with the computed secure control barrier certificate (S-CBC): B ( x ) = 0 . 1915 x 2 1 + 0 . 1868 x 1 x 2 − 0 . 144 x 1 + 0 . 1201 x 2 2 + 0 . 1239 x 2 + 0 . 16. The regions with red boundaries ( X 1 , X 2 , X 3 ) denote obstacles in the environment. X 0 is the set from which the agent starts at time 0. The contours show the values of the S-CBC of degree 2 ranging from 1 to 100. some part of X 1 . A possible reason is that when solving for the second condition in Proposition 1, we work with the union of the sets X 1 , X 2 , and X 3 , which may lead to a conservative estimate of the S-CBC . From Theorem 2 and the computed value of δ , we have that sup µ ( d ) inf µ ( a ) P x 0 µ ( d ) , µ ( a ) { L ( x N ) | = φ } ≥ 0 . 9922 . This bound is conservative in the sense that we consider defender inputs u d for only the extreme values of u a = − 1 and u a = 1. However , for the dynamics in Equation (20), if the last inequality in Proposition 1 is non-negative for both u a = − 1 and u a = 1, then for any u a ∈ [ − 1 , 1], this quantity will be non-negative. Determining methods to explicitly compute a defender policy and considering S-CBCs of higher degree is an area of future research. 7 Conclusion This paper introduced a new class of barrier certificates to provide probabilistic guarantees on the satisfaction of temporal logic specifications for CPSs that may be affected by the actions of an intelligent adversary . W e presented a solution to the problem of maximizing the probability of satisfying a temporal logic speci- fication in the presence of an adversary . The interaction between the CPS and adversary w as modeled as a discrete-time dynamic stochastic game with the CPS as defender . The evolution of the state of the game was influenced jointly by the actions of both players. A dynamic programming based approach was used to syn- thesize a policy for the defender in order to maximize this satisfaction probability under any adversary policy . W e introduced secure control barrier certificates, an entity that allowed us to determine a lower bound on the satisfaction probability . The S-CBC was explicitly computed for a certain class of dynamics using sum-of- squares optimization. An example illustrated our approach. Our example may have resulted in conservative bounds for the satisfaction probabilities since we restrict our focus to barrier certificates that are second degree polynomials and to stationary policies for the two agents. Future work will seek to study conditions under which possibly more effective non-stationary agent policies and higher degree S-CBCs can be deployed to solve the problem. A second interesting problem over a finite time-horizon is to investigate if explicit time bounds can be enforced on the temporal logic formula. An example of such a property is that the agent is required to reach a subset of states of the system between 3 and 5 minutes. This formula cannot be encoded in L TL, but there are other temporal logic frameworks like metric interval temporal logic [4] or signal temporal logic [31] that will allow us to express it. W e propose to study the case when the system will have to satisfy other kinds of timed temporal specifications [12] in the presence of an adversary in dynamic environments. References 1. Abate, A., Prandini, M., Lygeros, J ., Sastry , S .: Probabilistic reachability and safety for controlled discrete time stochastic hybrid systems. Automatica 44 (11), 2724–2734 (2008) 2. Ahmadi, M., Jansen, N ., W u, B., T opcu, U .: Control theory meets POMDPs: A hybrid systems approach. arXiv preprint arXiv:1905.08095 (2019) 3. Ahmadi, M., W u, B ., Lin, H., Topcu, U .: Privacy verification in POMDPs via barrier certificates. In: IEEE Conference on Decision and Control. pp . 5610–5615 (2018) 4. Alur , R., F eder , T ., Henzinger , T .A.: The benefits of relaxing punctuality . Journal of the ACM 43 (1), 116–146 (1996) 5. Ames, A.D ., Coogan, S ., Egerstedt, M., Notomista, G ., Sreenath, K., T abuada, P .: Con- trol barrier functions: Theory and applications. In: Proceedings of the European Con- trol Conference (2019) 6. Ames, A.D., Xu, X., Grizzle , J .W ., T abuada, P .: Control barrier function based quadratic programs for safety critical systems. IEEE Transactions on Automatic Control 62 (8), 3861–3876 (2016) 7. Ba ¸ sar , T ., Olsder , G .J .: Dynamic noncooperative game theory , vol. 23. SIAM (1999) 8. Baheti, R., Gill, H.: Cyber-physical systems. The Impact of Control T echnology 12 (1), 161–166 (2011) 9. Baier , C., Katoen, J .P .: Principles of Model Checking. MIT Press (2008) 10. Belta, C., Y ordanov , B., Gol, E.A.: F ormal methods for discrete-time dynamical sys- tems, vol. 89. Springer (2017) 11. Bertsekas, D .P .: Dynamic Programming and Optimal Control 4th Edition, V olumes I and II. Athena Scientific (2015) 12. Bouyer , P ., Laroussinie, F ., Markey , N ., Ouaknine, J ., W orrell, J .: Timed temporal log- ics. In: Models , Algorithms , Logics and T ools , pp. 211–230. Springer (2017) 13. Breton, M., Alj, A., Haurie , A.: Sequential Stac kelberg equilibria in two-person games . Journal of Optimization Theory and Applications 59 (1), 71–97 (1988) 14. Chow , C.S ., Tsitsiklis, J .N.: An optimal one-way multigrid algorithm for discrete-time stochastic control. IEEE Transactions on Automatic Control 36 (8), 898–914 (1991) 15. Cimatti, A., Clarke, E., Giunchiglia, F ., Roveri, M.: Nusmv: A new symbolic model verifier . In: International Conference on Computer Aided V erification. pp. 495–499. Springer (1999) 16. De Giacomo, G., V ardi, M.: Synthesis for L TL and LDL on finite traces. In: Interna- tional Joint Conference on Artificial Intelligence . vol. 15, pp. 1558–1564 (2015) 17. De Giacomo, G., V ardi, M.Y .: Linear temporal logic and linear dynamic logic on finite traces. In: International Joint Conference on Artificial Intelligence. pp. 854–860 (2013) 18. Ding, J ., Kamgarpour , M., Summers , S ., Abate, A., L ygeros, J ., T omlin, C .: A stoc hastic games framework for verification and control of discrete time stochastic hybrid sys- tems. Automatica 49 (9), 2665–2674 (2013) 19. Ding, X., Smith, S .L., Belta, C ., Rus , D .: Optimal control of MDPs with linear temporal logic constraints. IEEE Transactions on Automatic Control 59 (5), 1244–1257 (2014) 20. F arwell, J .P ., Rohozinski, R.: Stuxnet and the future of cyber war . Survival 53 (1), 23– 40 (2011) 21. Gordon, G.J .: Approximate solutions to Markov decision processes. Tec h. rep., School of Computer Science, Carnegie-Mellon University , Pittsburgh, P A (1999) 22. Jagtap , P ., Soudjani, S., Zamani, M.: T emporal logic verification of stochastic systems using barrier certificates. In: International Symposium on Automated T echnology for V erification and Analysis. pp . 177–193. Springer (2018) 23. Jagtap , P ., Soudjani, S ., Zamani, M.: F ormal synthesis of stochastic systems via control barrier certificates. arXiv preprint arXiv:1905.04585 (2019) 24. Kolathaya, S ., Ames , A.D .: Input-to-state safety with control barrier functions . Control Systems Letters 3 (1), 108–113 (2018) 25. K ˇ retínsk ` y, J ., Meggendorfer , T ., Sickert, S., Ziegler , C.: Rabinizer 4: From L TL to your favourite deterministic automaton. In: International Conference on Computer Aided V erification. pp. 567–577. Springer (2018) 26. Kupferman, O., V ardi, M.: Model checking of safety properties. In: International Con- ference on Computer Aided V erification. pp. 172–183. Springer (1999) 27. Kushner , H.J .: Stochastic Stability and Control. Academic Press (1967) 28. Kwiatkowska, M., Norman, G ., P arker , D .: Prism 4.0: Verification of probabilistic real- time systems. In: International Conference on Computer Aided V erification. pp. 585– 591. Springer (2011) 29. Lafferriere, G., Pappas , G .J ., Y ovine, S .: Symbolic reachability computation for families of linear vector fields. J ournal of Symbolic Computation 32 (3), 231–253 (2001) 30. Lindemann, L., Dimarogonas , D .V .: Control barrier functions for signal temporal logic tasks. IEEE Control Systems Letters 3 (1), 96–101 (2019) 31. Maler , O., Nickovic, D .: Monitoring temporal properties of continuous signals. In: F or- mal T echniques , Modelling and Analysis of Timed and F ault-T olerant Systems, pp. 152–166. Springer (2004) 32. Niu, L., Clark, A.: Secure control under L TL constraints. In: IEEE American Control Conference. pp . 3544–3551 (2018) 33. Niu, L., Li, Z., Clark, A.: LQG reference trac king with safety and reachability guaran- tees under false data injection attacks . In: IEEE American Control Conference (2019) 34. P arrilo, P .A.: Semidefinite programming relaxations for semialgebraic problems. Mathematical Programming 96 (2), 293–320 (2003) 35. P ola, G ., Manes , C ., van der Schaft, A.J ., Di Benedetto , M.D .: Bisimulation equivalence of discrete-time stochastic linear control systems. IEEE Transactions on Automatic Control 63 (7), 1897–1912 (2017) 36. Prajna, S ., J adbabaie, A., Pappas , G.J .: A framework for worst-case and stochastic safety verification using barrier certificates . Transactions on Automatic Control 52 (8), 1415–1428 (2007) 37. Prajna, S., Papachristodoulou, A., Parrilo , P .A.: Introducing SOSTOOLS: A general purpose sum of squares programming solver . In: IEEE Conference on Decision and Control. vol. 1, pp. 741–746 (2002) 38. Puterman, M.L.: Markov decision processes: Discrete stochastic dynamic program- ming. J ohn Wiley & Sons (2014) 39. Ramasubramanian, B ., Clark, A., Bushnell, L., Poovendran, R.: Secure control under partial observability with temporal logic constraints . In: IEEE American Control Con- ference (2019) 40. Saha, I., Ramaithitima, R., Kumar , V ., P appas, G .J ., Seshia, S .A.: Automated composi- tion of motion primitives for multi-robot systems from safe L TL specifications . In: Proc. International Conference on Intelligent Robots and Systems. pp . 1525–1532 (2014) 41. Santoyo, C ., Dutreix, M., Coogan, S.: V erification and control for finite-time safety of stochastic systems via barrier functions. In: IEEE Conference on Control T echnology and Applications (2019) 42. Sharan, R., Burdick, J .: Finite state control of POMDPs with L TL specifications. In: IEEE American Control Conference. pp . 501–508 (2014) 43. Shoukry , Y ., Martin, P ., T abuada, P ., Srivastava, M.: Non-invasive spoofing attacks for anti-lock braking systems. In: International W orkshop on Cryptographic Hardware and Embedded Systems. pp . 55–72. Springer (2013) 44. Slay , J ., Miller , M.: Lessons learned from the Maroochy w ater breach. In: International Conference on Critical Infrastructure Protection. pp. 73–82. Springer (2007) 45. Steinhardt, J ., T edrake, R.: Finite-time regional verification of stochastic non-linear systems. The International J ournal of Robotics Research 31 (7), 901–923 (2012) 46. Sullivan, J .E., Kamensky , D .: How cyber -attacks in Ukraine show the vulnerability of the US power grid. The Electricity Journal 30 (3), 30–35 (2017) 47. T arjan, R.: Depth-first search and linear graph algorithms . SIAM J ournal on Comput- ing 1 (2), 146–160 (1972) 48. T oh, K.C., T odd, M.J ., Tütüncü, R.H.: SDPT3: A MATLAB software package for semidefinite programming. Optimization methods and softw are 11 , 545–581 (1999) 49. W ongpiromsarn, T ., T opcu, U ., Lamperski, A.: Automata theory meets barrier certifi- cates: T emporal logic verification of nonlinear systems. IEEE Transactions on Auto- matic Control 61 (11), 3344–3355 (2015) 50. Y ang, I.: A dynamic game approach to distributionally robust safety specifications for stochastic systems . Automatica 94 , 94–101 (2018)