Experimental Aspects of Synthesis

Johannes Reich and Bernd Finkbeiner (Eds): International W orkshop on Interactions, Games and Protocols (iWIGP) EPTCS 50, 2011, pp. 1–16, doi:10.4204/EPTCS.50.1 c  R. Ehlers This work is licensed under the Creati ve Common s Attribution-Nonco mmercial-No Deriv ati ve W ork s License. Experime ntal Aspects of Synthesis R ¨ udiger Ehlers Reactiv e systems grou p, Saarland Univ ersity W e discuss the problem of experimentally ev aluating linear-time temporal logic ( L TL) syn thesis tools for r eactiv e systems. W e first sur vey pre vious suc h work for th e cu rrently pu blicly available synthesis tools, and then draw conclusions by deri ving useful schemes for future such e valuations. In pa rticular, we explain why previous tools have incompatib le scopes and semantics and provide a framework that re duces the impact of this problem for future experimen tal co mparisons of such tools. Furthermo re, we discuss which difficulties the complex workflows that be gin to appear in modern synth esis too ls ind uce on experimental ev aluations and give answers to the question how convincing such evaluations can still be perform ed in such a setting. 1 Introd uction The problem of synthesizi ng reactiv e systems fro m lin ear -time temporal logic (L T L) specificati ons [58, 1, 59, 46, 73] has attracted many researcher s in the past, leading to a tremendous amount of result s in this area. Broadly , wo rks contri b uting to the progress of its solution can be classified into two sorts. On the the ory side, major bre akthro ughs ha ve be en obtaine d by establish ing the 2EXPTIME- hardne ss of this probl em [6 0, 74, 25], along with asymptoti cally optimal au tomata-t heoret ic constr uc- tions for solvin g it [59, 1, 73]. Recent works are concerned with making these construc tions easier [49, 65, 5 7] or en hancin g the scope of th e algorithms an d hardness-re sults kno wn to, e.g., distr ib uted systems [48, 24]. On the pra ctical side, m any works deal w ith the constructi on of sophisticate d algorit hms that aim at impro ving the scalability of current synth esis techni ques [38, 66, 22, 2 3, 5 7, 1 6]. While th e 2EXPTIME- hardne ss of the L TL synthesis pro blem induces a limit on the effe cti v eness of any approac h, this line of researc h is moti v ated by the observ ation that “typi cal“ specificatio ns in practice ha ve a structu re that can be expl oited [44, 67, 52, 23, 66, 57, 16, 45]. While many w orks fall into both cat ego ries, contrib utions to the latter sort typically contain proofs of the usefulne ss of the proposed techn iques obtained by expe rimental ly ev aluating a prototy pe imple- mentatio n. This is commonly done by taking some example specificatio ns (the so-calle d benchmarks ) and sho wing that the prototy pe is able to handle these in reasonable time. The situati on is similar to the one for the problem of satisfia bility (SA T) testing (see, e.g., [4]), where despite its NP-completenes s, an acti v e area of research has emerg ed, w itness ing its progress by the f act th at pract ical proble ms with millio ns of v ariables are no wadays rou tinely solv ed by modern SA T solv ers. O ne of the key factors for thi s succe ss is the possib ility to perform meanin gful benchmar king, which dri ves the de v elopmen t of ne w solu tion heuris tics into tho se direct ions that appear to be most promisin g. Thousands of example prob lem instances can easily be used to obtain, optimise and test ne w approa ches. As a result, th e annual SA T solv ing c ompetiti ons t ypical ly dra w a lot of interest, fo r example 19 submissio ns to the main track in the 2010 SA T -Race [55]. For the sy nthesi s of rea cti v e systems from linear -time specification s, howe v er , there a ppears t o be f ar less interest in tools. At the time of writing, to the best of our knowled ge, there are only four publicl y 2 Experiment al as pects of synthe sis a v ailabl e synth esis tools 1 , namely A N Z U [9, 39, 7, 8 ], L I L Y [37, 38], A C AC I A [13, 22, 23], and U N B E A S T [17, 16]. Equally unfortu nate, the amount of benchmark s av ailab le is rather low . One can identif y (at least) three reason s for this state: 1. A factor contrib uting to this differe nce is the fact that while rudimentary SA T solve rs can be written in the order of hours, creating eve n a simple syn thesis tool requires the implementa tion of comparabl y comple x ope ration s. As an examp le, for ap proach es worki ng with determini stic automata , a constr uction simil ar to Safra’ s determinisati on proced ure [6 4] needs to be perf ormed, which has been ar gued to be notori ously complex to implement [35, 34, 38, 44, 45]. 2. A s a second reason , the publicatio n schemes of the fo rmal method s and SA T communities are dif feren t. While the latter appreciates work whose primary concern is to improve the scalabil ity of curre nt S A T solvin g techniques (see, e.g., [42, 54, 28] for recent such publicati ons from 2010), works in the formal method s area are typicall y b uilt around some appealin g ne w conc ept, which is mostly only ev aluated briefly on some pr ototyp e implementation (see, e.g., [29] for a pointer to a typic al such a case in the area of sof tware model check ing, or simply compare recen t practical synthe sis papers [23, 16, 22, 38, 57, 66, 67]). Arg uably , on e of the main reas ons for this dif feren ce is the fact that the roo ts of formal methods lie in theoretical computer scie nce, where, gi ven technical correctness and suf ficient general style of writing, the main merits of a paper are seen in the signi ficance of the conceptu al contrib ution to the field [56]. As a result, papers that propos e impro vemen ts to current techniqu es that lack a major theore tical insig ht ha v e a small chance of being accepted at major conferenc es ev en if the propo sed techniq ues lead to significant speed -ups in the synthesis proc ess. Consequ ently , time is typically onl y in ves ted in w riting synthesis tools after a new idea has been dev eloped that is both theoretical ly compell ing and giv es the impression that it will significantly impro ve upon the perfor mance of pre vious technique s. 3. T hird, e ven if time has been in veste d in writing a synthesis tool, a new technique still has to be sho wn to be competiti ve with earlier techn iques . T ypically , benchmarkin g is used for this purpo se. In the scope of synthes is, ho we ve r , it can be obs erv ed that this is by no means a tri vial task – all four currently av ailable synthesis tools ha ve dif feren t scopes and sema ntics. Also, the amou nt of bench marks av aila ble is extremel y low and the benchmark s i n pre viou s e v aluat ions hav e often been re written to be compatible to the improv ements proposed. This puts a hi gh b urden on the quality of future w orks in th is area: compa rabilit y to p re vious works must be maintained in order to o btain a high credibility of the experimenta l e v alua tion. At the sa me time, as improv ements to syn thesis techni ques often introduce additional details that m ust be tak en care of in the ev alua tion (e.g., ha ving two semi-alg orithms running in parallel in [22, 16] or the assumptio n dropping heuristic from [23]), it is ve ry hard to produce an appealin g e valua tion w ithout spend ing too much space in a public ation on the details. Recently , the first of these problems has been attenuated by the a v ailabi lity of good L TL-to-B ¨ uchi transla tion tools [63, 27, 14] and determin isatio n and optimisatio n tools for ω -automat a [40, 15, 18] 1 For the scope of this paper , we exclude all tools t hat aim at pure game solving, as here, the synthesis functionalities and the possibility to start from linear-time t emporal logic is mi ssing (which is t ypically not merely a preprocessing step). There is another tool called R A T S Y [6], which we excluded as its synthesis functionality is mainly a reinterpretation of A N Z U (plus some preliminary implementation of the bounded synthesis approach [65, 22]) and the tool aims at pro viding an en vironment for specification eng ineering rather than being on ly a synthesis tool. Consequ ently , no experimental e valuation of the synthesis performance has been giv en in [6]. The J T LV [61] scripting env ironment that also has synthesis procedures has been exclude d as no stand-alone synthesis tool exists and benchm arks comparisons are not ava ilable. R. Ehlers 3 on the one hand, and the av ailability of efficient binary decision diagram (BDD) libraries [68, 10] and satisfiabi lity modulo theory (SMT) or S A T -solv ers [4] as reasoni ng back bones on the other hand. Thus, de ve lopers of ne w synthesi s tools can bu ild their implementatio ns on top of such pre vio us wo rk. The second problem will hopefull y lo se impact over time when more intere st in the practical side of reacti ve system synthesi s is a roused . In this paper , we approach the remain ing third problem by gi ving b oth insights why expe rimental e v aluati ons in the conte xt of synthes is are notori ously harder than, e.g., in the SA T contex t, as well as propo sing ”standa rdised “ ev alua tion sche mes for synthesis too ls that aim at simp lifying fu rther work in this area. W e hope that our disc ussion help s int erested obs erve rs of the adv ances in the practi cal ap- proach es to synthesis (by pro vidi ng a surv e y on the problem of ev aluating synthesis tools , with spe cial consid eratio n of work already don e in this area) as well as auth ors of future synthesis tool s (by givin g in- spirati ons and i n partic ular justificat ion for the choice of their exp erimenta l settin gs) and p aper re vie w ers in the field (by expl aining the difficul ties of performing an experimen tal ev aluat ion for synthes is tool s). W e start by gi ving a definitio n of the L TL (open) synthesi s proble m in Section 2. In S ection 3, we re vie w the synthesis approa ches of the synthes is tools that were pu blicly a v ailable at the time o f writin g. Afterwar ds, we gi v e some observ atio ns on thes e approa ches (and their e xperi mental ev aluations). In Section 5, we analyse the obse rv ation s and prop ose a frame work for fut ure synthesi s tool exp erimenta l e v aluati ons. W e con clude with a summary . 2 The L TL open synthesis pr oblem W e start by giv ing a problem descriptio n of reacti ve system synthe sis that focusses on those aspects that requir e speci al atten tion w hen comparing syn thesis ap proach es. Formally , a synthesis problem instan ce is a triple h AP I , AP O , ψ i , where A P I is a set of input ato mic proposition s, AP O is a set of output atomic propo sitions , and ψ is a formula in line ar -time temporal l ogic (L TL ) [58] ov er AP I ⊎ A P O . For the scope of this paper , w e denote the L TL temporal operators for ”globally”, ”finally” and ”next- time” by G , F , and X . For the ease of reading , we sometimes call the atomic propositi ons simply v ariab les or bits. W e say that a trip le h AP I , AP O , ψ i rep resen ts a re alisab le specificat ion in the Mealy- type se mantics if there ex ists some func tion f : ( 2 AP I ) + → 2 AP O such that for all w = w 0 w 1 . . . ∈ AP ω I , we ha ve ( w 0 ∪ f ( w 0 )) , ( w 1 ∪ f ( w 0 w 1 )) , ( w 2 ∪ f ( w 0 w 1 w 2 )) , . . . | = ψ . Like wise, we say tha t h AP I , AP O , ψ i is rea lisable in the Moor e-typ e semantics if there exists some function f : ( 2 AP I ) ∗ → 2 AP O such that for all w = w 0 w 1 . . . ∈ AP ω I , we ha ve ( w 0 ∪ f ( ε )) , ( w 1 ∪ f ( w 0 )) , ( w 2 ∪ f ( w 0 w 1 )) , . . . | = ψ for ε denoti ng the empty word. Specification s that are not realisable are called unrealisa ble in the re specti ve se mantics. T ypically , realisab ility che cking is performed by build ing a game between a system player and an en vir onment player . In th is setting, a funct ion f satisf ying th e constraint s stated abov e is called a winning stra te gy . Details on the game-based vie w to synthesis can be found in [32]. It is well-kno wn that whene ver there exists some winning strategy for one of the semantics abov e, there also exists a finite repres entati on of it. For the Mealy-type semantics, thi s representat ion is typically gi ve n as a Mealy automato n, whereas for the Moore-type semantics , M oore automa ta serve this purpos e (see, e.g., [53]). Intuiti vely , the Mealy- and M oore-t ype semantic s diff er in the order of input and output. As an exa mple, for the L T L formula ψ = G ( r ↔ g ) , the specification h{ r } , { g } , ψ i is realisab le for the Mealy- type semantic s bu t not for the Moore-type semanti cs. The reaso n is th at in the Meal y-type s emantics , the system already kno ws the input in the respec ti ve computation cycle when ha ving to choose an output, whereas in the Moor e-type semantics, the roles are sw apped and thus the system has to gue ss w hethe r r 4 Experiment al as pects of synthe sis is set or not when choos ing whether g should be set. 3 Synthesis tools W e briefly recapitulate the ideas behind the synthesis tools A N Z U [39], L I L Y [37, 38], A C AC I A [13, 22, 23], and U N B E A S T [17, 16]. W e use the terminolo gies from the respecti ve papers and refer the reader not famili ar w ith the terms used here after to them. 3.1 A N Z U Scope: This too l implements the con cept of gener alised reacti vity(1 ) [57] synth esis (abbre viat ed as GR(1) synthesis) in the Mealy-type seman tics. Here, the specificati on is restricte d to be o f the form ( a 1 ∧ a 2 ∧ . . . ∧ a n ) → ( g 1 ∧ g 2 ∧ . . . ∧ g m ) for some sets of assu mptions { a 1 , . . . , a n } and guara ntees { g 1 , . . . , g m } . Every assu mption is of one of the follo wing forms: (1) ψ I (2) G ( ψ → X ( ψ I )) (3) GF ( ψ ) where ψ is an L T L-formula over A P I ∪ A P O free of temporal operat ors and ψ I is an L T L-formula over AP I free of temporal operator s . Like wise, all guarantees are of one of the followin g forms: (1) ψ O (2) G ( ψ → X ( ψ O )) (3) GF ( ψ ) where ψ O is an L TL-formula over AP O free of temporal op erator s. A N Z U requir es the assumptions and guaran tees to be in Pr operty Specificat ion Langua ge (PSL) [19] sy ntax and checks for str ong rea lis- ability of the giv en ove rall specification , where unlik e in normal realisability ch ecking , safe ty guarantee violat ions are not tolerat ed in cases in which a safety assumpt ion violati on has not yet been witnessed b ut the system has a strate gy to ens ure that the ov erall input and outp ut will no t satisf y all assumptions (see, e.g., [41]). Specificat ions of the ( V assumpti ons ) → ( V guaran tees ) form, as used in A N Z U , typi- cally occur in cases in which a part of a large r system is to be synthesized , where the set of assumptions repres ents the beha viour the part of the system to be synthesiz ed can assu me about the other parts of the system whereas the guarantee s describe the requirements on the beha viour of the part of the system to be synthes ized. T echniques: A N Z U implements ge neralis ed react i vity(1 ) synthesis [5 7] in a symbolic manne r using binary dec ision diagrams (BDDs) [ 10, 68] as reaso ning backbo ne. Here, a sy nthesi s game is b uil t whose state space consists of all v ariabl e valu ation s to the input and output atomic proposition s. The L TL assumpti ons and guarant ees of the forms ψ I and ψ O are encoded into the set of initial positions of the game, whereas assumptions and guaran tees of the form G ( ψ → X ( ψ I )) and G ( ψ → X ( ψ O )) are encod ed into its transitio n relation (des cribing the possibl e mov es of the pla yers). Then, a symbolic algori thm is used in which the system player tries to satisfy the specification ( a ′ 1 ∧ a ′ 2 ∧ . . . ∧ a ′ n ′ ) → ( g ′ 1 ∧ g ′ 2 ∧ . . . ∧ g ′ m ′ ) in this so-called game ar ena (consi sting of the game positio ns and the trans ition relatio n), where { a ′ 1 , . . . , a ′ n ′ } are th e a ssumptio ns of the form GF ( ψ ) and { g ′ 1 , . . . , g ′ m ′ } are th e g uaran tees of the form GF ( ψ ) . Implement ations are extrac ted by b uilding circuits for comp uting the out puts from R. Ehlers 5 the BDD represen tation of the w inning state set while performin g a care -set optimisatio n to the BDD after ev ery step [7, 8]. Experimental e va luation: The usefu lness of the tool A N Z U has been sh o wn on tw o case studies: an AMB A A HB arbiter [2] specification (simplified by lea vin g out b us splits and early b urst terminations) and a generalised Buffer (GenBuf) control ler tha t has been described by IBM for tuto rial purposes [36]. Both ca se studi es conta in se ver al assump tions and gu arante es and are scalab le by the n umbers of clients. 3.2 L I L Y Scope: The too l L I L Y accepts arbitrary L T L for mulas in PSL syntax as spe cification s. If multiple L TL formulas are found in the input file, they are treated in a conjuncti ve manner , e xcept for those that are preced ed by an ass ume key word, which are used as assumptio ns for the overal l specification . T echniques: L I L Y implements th e optimisations to the Safraless synthesis [49 ] approach presented in [38]. In the first ste p, the ne gation of the specification is con verted to a nod e-labe lled nonde terminist ic B ¨ uchi word (NBW) automato n using the L TL-to-B ¨ uchi transl ator W R I N G [69]. The B ¨ uchi automaton is then con v erted to a uni vers al co-B ¨ uchi tree automaton (UCT) that checks for the satisf action of the origin al specification along all computation tree paths, which is in turn tested for emptines s using a constr uction pro posed b y Kupferman and V ardi [4 9], u tilisin g alt ernati ng weak tree automata (A WT) and non-d eterminis tic B ¨ uchi tree (NBT) au tomata. Jobstmann and B loem [38] add vario us optimisa tions to these steps. The con version of the UCT to the A WT is parametrise d by some constan t k , whi ch influences the size of the NBT prod uced and thus the running time of the ov erall algorith m. While low v alues for k typica lly suf fice for re alisabl e specifica tions in prac tice, a relati v ely lar ge va lue of k , exp onenti al in the size of the N BW (or , alter nati v ely , doubly-e xpon ential in the length of the L TL specificatio n), is neede d to hav e the emptines s of the langua ge of the NBT imply the emptiness of the UC T language (e xcept if the UCT turn s out to be weak), and thus p rov e u nrealis ability of the giv en spe cification . T o av oid this problem, L I L Y can also be run in a special unrealisab ility dete ction mode. In this case, it checks the realisa bility o f the neg ated specificati on with swappe d inputs and outpu ts (and a slight modification of the resulti ng spe cification to con vert its Mealy-type se mantics to Moore-type again). Then, for unrealisabl e specifica tions, on ly a small va lue of k is typically needed in practice to identify them as such. Experimental eva luation: In [38], th e performance of L I LY is e v alua ted on some spec ifications writ- ten by the authors of tha t pape r , representing mostly arbite r varia tions and traf fic light con troller s. The e v aluati on is focussed on proving that the optimisations proposed in that paper contrib ute significantly to ha ving lo w running times of the tool. 3.3 A C AC I A Scope: The tool A C AC I A has the same input syntax as L I L Y and also uses Moore-typ e semantics. There exist two ve rsions of A C A C I A . A C A C I A 2009 implements the techniques described in [22], w hile A C A C I A 2010 also implements t hose of [23]. The latter ve rsion includes s upport for making assumpt ions local to some set of guara ntees. The specifica tion then consists of a conjunc tion of sub-speci fications of the form ( V assumpti ons ) → ( V guaran tees ) . All assumptio ns and guarante es of the conjun cts are assumed to be gi ve n separatel y . 6 Experiment al as pects of synthe sis T echniques: A C AC I A is based on the conce pt of bounded synth esis [65, 22], a r efinement of the Safra- less synth esis techn iques proposed in [49]. Here, as in L I L Y , the specificatio n is first ne gated and then con vert ed to a node-lab elled B ¨ uch i automato n. A s L I LY , A C AC I A u ses W R I N G [69] for th is purpo se. Afterwar ds, the B ¨ uch i automato n is con verte d to a uni versal co-B ¨ uchi tree automato n (UC T) that checks for the sa tisf action of the spe cification along all pa ths of a computation tree. This UCT is used as a ba sis for build ing a series of synthesi s safety g ames, where for a successi vely increasi ng bou nd value k , f or e v- ery state q in the UCT , the maximum number of visits to re jecting states in the UCT from its initi al state along some path to q for the input/outpu t played in the game so far is encod ed into the game positi ons. Once one of these counters exce eds the valu e k , the game is lost for the system player . T he main idea of A C AC I A is to u se anti- chains as an efficie nt represen tation of th e fr ontier sets (i.e., pre-fixed po ints of winning pos itions ) occurring during th e safety game solvin g process . This represe ntatio n makes use of the fact that the set of pos sible future beha viou rs of the sys tem player in a positio n p 1 can only be lar ger than when being in a game pos ition p 2 if all counte rs in p 1 are less than or equa l to those in p 2 . Thus by storin g on ly states w hose counter vectors are not dominated by the counter vectors of other sta tes in th e pre-fix ed point during the solving process, redundanc ies can be a v oid ed. As in L I L Y , the va lue of k required to conclude the unreal isabili ty of a spec ification is ex ponent ial in the size of the U CT or doubly -ex ponen tial in the length of the L T L specificati on. The authors thus propo se to run A C AC I A two times in parallel, where in the first run, realisabil ity is check ed and in the second ru n, unreali sabilit y is tes ted. A s in L I L Y , in the la tter case, the specification is nega ted, a con vers ion between Mealy-type and Moore-typ e semantics tak es place, and the inputs and outputs are swapp ed. A C A C I A 2010 adds some additiona l features . Here, the game solving process is made composi- tional . Recall tha t in A C AC I A 2010, the s pecificati on is su ppose d to consist of a co njunct ion of sub- specifica tions of the form ( V assumpti ons ) → ( V guaran tees ) . In th is settin g, the safety games for the synthe sis process can be buil t separa tely , prel iminarily solved independ ently and finally composed on- the-fly duri ng the solving proce ss for the game repre sentin g the ov erall speci fication. A C A C I A 2010 furthe rmore ad ds the possibil ity to use the OT F U R mixed forward-bac kward g ame solvin g algorithm [50, 11] ins tead of the classical ba ckwar d safe ty ga me so lving algorithm. Finally , for cases in which the specification is only a single formula of the form ( V assumpti ons ) → ( V guaran tees ) , A C AC I A ca n re write this specificatio n int o the form V g ∈ guaran tees (( V assumpti ons ) → g ) in order to benefit from the composi tional algorithms implement ed. In th is ca se, an as sumptio n dropping heuris tic is use d to re mov e some assumption copi es in this formul a, which reduces the problem that the ass umption s are replicated for all guarante es in this setting. U sing the heuris tic makes the approac h ho wev er incomplete . Experimental ev aluatio n: In [22], the focus of th e exp erimenta l ev alua tion lies on prov ing that A C A - C I A improv es upon the performance of L I L Y , using the fact that the semantics are compatible. The author s of [22] sho w that on the examples from [38], using the anti-chai ns approach typ ically res ults in lo wer computation times and that the B ¨ uchi automato n b uild ing time surpris ingly domin ates the o ve rall synthe sis time. One of the example sp ecificatio ns is made scalabl e and it is prov en that the anti-chain s approa ch is much faster here. Anot her set of variat ions of one of the L I L Y ex amples is used as a further bench mark set. In [23 ], the 2010 version of A C A C I A is ev aluated with sev eral diffe rent choices for (1) whether game so lving sho uld be perfo rmed backwa rds or in a forward- backw ard manner , (2 ) wheth er monolithic or compositiona l synthesis should be performed, and (3) whether the assumptio n droppi ng heuris tic should be used (only in the compositional case). Apa rt from the benchmarks also used in [22], the R. Ehlers 7 genera lised Buf fer (GenBuf) controller [36] specificat ion that was also u sed for ben chmarki ng A N Z U has b een fo rmulated in a way such that the assu mptions to the en vironment are local to some se ts of guaran tees, such that compositio nal synthesis can be perfor med directly . This benc hmark is used to sho w the benefits of the composi tional approach. 3.4 U N B E A S T Scope: The tool U N B E A S T [17, 16] focusses on specification s of the form ( V assumpti ons ) → ( V guaran tees ) and uses Mealy-type semantics. By using an input language based on XML , incorrect presump tions by the user about prec edence s of temporal operator s in L T L ar e a v oided . The assumptio ns and guaran tees are giv en separatel y in the X ML input file. T echniques: The U N B E A S T tool implements the synthes is tech nique s pres ented in [65, 16]. The li- brary C U D D [68] is used for constructin g and manipu lating BDD s dur ing the synthesis process. The first step is to dete rmine w hich of the giv en assumptions an d guarantees are sa fety formulas. In order to d etect also simple cas es of patholog ical safety [47], this is done by comput ing an equi valent B ¨ uchi automaton using an external L TL-to-B ¨ uchi con verter such as L T L 2 B A [2 7] or S P OT ’ s LTL 2 T G BA [14], a nd e xamini ng whethe r all maximal strong ly connec ted componen ts in t he co mputed au tomaton do not hav e infinite non-acce pting pa ths. Special care is tak en of so-called of bounded look-ahea d saf ety formulas . In a s econd step, for th e set of bounded lo ok-ahe ad assu mptions and the set of such guarantees , safety automata for th eir res pecti ve conjunc tions are bui lt. Both of th em are repre sented in a symbolic way using BDDs. For the remaining safet y assumptio ns and guara ntees, safety automata are built by taking the B ¨ uchi automata co mputed in the pre vio us step and ap plying a subset constructio n for dete rminisati on in a symbolic manner . For the rema ining non-s afety parts of the specification, a combine d uni ver sal co-B ¨ uch i automaton is computed by calling the externa l L TL-to-B ¨ uchi tool agai n. In the next phase, the gi ve n specificatio n is check ed for realisabil ity . This is done almost as in A C A C I A 200 9, i.e., for a successi vely increasing so-called bound value , the bounded synthesi s ap proach [65, 22] is performed by build ing a safety automato n from the co-B ¨ uchi automaton for t he non-sa fety part of the sp ecificatio n and solving the safety ga mes induced by a spe cial pr oduct of the automata in volv ed [16]. Howe ver , instead of anti-chain s, BDDs are use d. Finally , if the specifica tion is found to be realis able (i.e., the game compu ted in the pre vio us phase is winnin g for the player repre sentin g the syste m to be synthesised ), the symboli c represen tation of the winning states of the system is used to compute a prototyp e impl ementatio n sa tisfyin g the specification in a fu lly symbolic way , using a sligh t simplifica tion of the algori thm from [43]. Howe v er , the implemen- tation s gene rated are typ ically relati vel y lar ge. As A C A C I A , to also detect un realisa ble speci fications , U N B E A S T needs to be run two times in paral lel. Experimental e val uation: U N B E A S T was e v aluat ed on the specifications defined in [38 ] a s well as on those giv en in [22]. The Moore-ty pe semantics from th ese exampl es hav e been adapted to the Mealy-ty pe semantic s of U N B E A S T by prefixing all occ urrenc es of input atomic prop osition s with a n L T L ne xt-t ime operat or (see, e.g., [38 ]). Addition ally , a scalable load balanci ng case study is presented in [16], ha ving a Mealy-type se- mantics. For comparison and usage with A C AC I A and L I LY , the examples ha ve been transformed to Moore-ty pe semantics by prefixing all occ urrenc es of ou tput atomic propo sition s with an L TL nex t-time operat or . 8 Experiment al as pects of synthe sis 4 Observations on the differ ences and similarities of the synthesis tools W e continue with a discus sion of the similarities and differ ences betwee n the sy nthesi s tools . The argu- ments to follo w form the foundat ion of the experimen tal ev alu ation frame works propo sed in S ection 5. 4.1 T he incomparable scopes & semantics of the tools When comparin g the tools co nsider ed in this pap er , it is striking that all four tools ha ve incompatib le specifica tion langua ges. Only A C AC I A 2009 and L I LY hav e the same input specification format (how- e ver , A C AC I A 2010 ad ds local assumptions to th e in put lan guage which cannot be interpreted correctly by L I L Y ). It is fair to raise the question why this is th e ca se, gi ven the fac t that the number of benchmark sets for synthesis is rather lo w , so one would expect that the scopes and semantics are compatible in order to ha ve as many benchmar ks a v ailab le as possible. In this paper , we conjecture that the reason for this situati on is tha t the scopes of the tools are str ongly adapte d to the techni ques implemented , b ut whene v er the choice does not matter , as close to the literature as possible. W e begin our discussio n of this observ atio n with the tool A N Z U . The generalised react i vity(1 ) syn- thesis techniq ue is implementable in both Mealy and Moore semantics. The tool A N Z U uses Mealy semantic s, as the d escrip tion of the synthe sis algo rithm in [57]. T he assumptions and guarantee s allo wed are precise ly those that can be processed by the algorithm without gi ving up the idea that the state space of the underlying game is the set of input and ou tput variab le v alu ations . This can be seen from the fact that fro m the formula types gi ven in S ection 3.1, type (1) is only an initial state co nditio n, and typ e (2) can be enc oded int o the tran sition rel ation of a game with such a structure. Formula s of ty pe (3) are precis ely those that can then be giv en to the actua l solving process as liv enes s parameters. In contrast to A N Z U , L I LY uses Moore-ty pe semantics and some PSL-like input file syntax. Since L I L Y bases on tree automaton te chniqu es, this is not surprising: using a Mealy -type seman tics in the conte xt of tree automata would require that the labellin g of the initial node of a computation tree (that is either accept ed or rejecte d by the tree automat on) is ignore d, as the nod e labels repr esent the outpu t of the system. On a theoretic al lev el, such a definitio n woul d look unnecessar ily awkward, which is why the Moore-ty pe semantics are usually preferred in this context . As L I L Y , A C AC I A uses a Moore-typ e semantics an d the same syn tax as L I L Y . According to [22], the author s wanted to keep A C A C I A 2009 compar able to L I L Y . An a dditio nal reason f or k eepin g the Moor e- type semantics is the better applic ability of the results: specification s that are found to be realisab le in the Moore-type semantic s are also realisable in the Mealy-ty pe semantics, but not vice versa. Only in A C A C I A 2010, the input language is extended in order to accommoda te the ne w features proposed in [23]. U N B E A S T , on the other hand, uses a M ealy-t ype semantics and has its own XML-based input file format. In [16], it has been ar gued that spec ification s often be come shorter and thus the (ed ge-lab elled) B ¨ uchi automata become smaller in the Mea ly setting, which is beneficial for a BDD -based appro ach. For example, when specifying some immedia te outp ut consequ ences of some input such as G ( r → g ) for some input set { r } and outpu t set { g } , taking the Moore seman tics would requ ire the introduc tion of a nex t-time operator into the formula, which would be reflected in the automato n size. The XML -based input language has been used in ord er to circ umve nt the neces sity fo r a co mplicate d formula parser , bu t also to make the ope rator precedence s ex plicit. R. Ehlers 9 4.2 Comp arability of the examples The ar gument s fro m the prece ding subsection explain why the sc opes and semantics of the tools are dif - ferent. Ne vert heless , it does not exp lain why dif ferent speci fication sets hav e been used, as benchmarks for A N Z U could be con v erted to benchmarks fo r the other tools and the co n version between Mealy- and Moore-ty pe semantics is rather simple. S till, the only case in which be nchmark s were con vert ed for an exp erimenta l ev aluation was in [16], where L I LY ’ s and A C AC I A ’ s ex amples were used for e valuat ing U N B E A S T (and vice ve rsa). In some cas es, benchmarks ha v e been re written , e.g., the IB M gen eralise d b uf fer spec ification, which was used to e v aluate A N Z U , has been altere d in [23] to a form in w hich the assumpti ons were made local. Also , there are tw o other publicatio ns [66, 51] reporting on experiment al results for synth esis ap proach es using generalize d parity games. In bo th of them, the feasi bility of their approa ches is shown us ing dif fere nt reformulations of the AMBA arbiter example tha t was also used for A N Z U . An expla nation for this fact was gi v en in [67, 66 , 5]. In fact, the GR(1) synthesis approach imple- mented in A N Z U can accommod ate all types of assumption s and guara ntees that are repres entable as determin istic B ¨ uch i automat a (DB A) [5, 15]. In order to fit into the input language of A N Z U , howe ver , the output bit set of the sys tem to be synthes ized has to be ex tended by state bits of the auto maton. Somenzi and Sohai l coined the term “pr e-synth esis” for such an encoding , as con v erting an assumption or guarantee to such an automat on and encoding it into some outp ut bits in a good way is a problem on its own for BDD-based techniq ues (s ee, e.g., [31, 26]). Thus, a lot of ef fort has been put into a good reformul ation of the problem descriptio n before checking realisability . An equiv alent app roach to usi ng DB As is to introduce so-called auxiliary signals (or auxiliary variab les) into the de sign [30, 8, 7]. It has been note d that rewritin g a spe cification using differe nt signals can significantly speed up the synt hesis proces s [30, 8] and for the AMBA AHB specifica tion, this has also been done. As a consequenc e, it is not surpris ing th at a specificati on for which pre-synt hesis wa s perfor med and that has been optimized to ward s A N Z U , neither the auth ors of A C AC I A nor U N B E A S T (n or the aut hors of the works using gen- eralize d parity automata [66, 51]) used the AMBA A HB arbiter spe cification s in the form pro vided with the A N Z U tool for comparison s. W ith respect to the fact that the IBM Generalise d Buf fer example has been altered for usage with A C A C I A 2010, the situatio n is similar: in the original specificat ion, the assumptions were not localised; defining the scope of the assumptions was simply not an issue in this case. As soon as techniques are introd uced that can m ake u se of such local assumption s, the s ituatio n changes. 4.3 Comp lexity of the workflo ws Except for A N Z U , the workflo ws, i.e., the numbers and orders of computation steps in the realisability checki ng process, of the tools discussed here are rathe r complicate d. L I LY and A C AC I A 2009 first con vert the specificati on to a univ ersa l B ¨ uchi automat on and then perform, for some success i vel y increasin g bound v alue, a r ealisab ility check o ve r t his a utomaton . In ord er to also detec t unreal isable specificat ions, the check must additiona lly be ran for the nega ted specification with a con version between the two semantic s types in parallel. The workflow of U N B E A S T is similar . In contr ast to man y other formal methods experimen tal e v aluat ions, this w hole process is relati v ely compli cated and might eas ily appear less compelli ng than more simple schemes that are used in, for example, SA T solv ers. It is fair to conjecture that future workflows will ev en be more compli cated. T ake for example, A N Z U , which has a relati v ely straigh t-forw ard workflow . As it h as been a r gued that th e gen eralise d reactiv ity(1) synthe sis approa ch that is used in this tool could handle all assumption s and guarantees that are repre- 10 Experiment al as pects of synthe sis sentab le by determinis tic B ¨ uch i a utomata, de v elopin g a preproces sor that takes L TL specificatio ns of this kind and produces equi v alent A N Z U specification s appear s to be wo rthwhile to write. Howe v er , such a prepro cessor w ould hav e a very complicat ed workflo w . A fter con verting the assumption s and guaran- tees to B ¨ uchi automata, these ha ve to be determinised (whene v er possibl e), usin g an extern al tool like L T L 2 D S T A R [4 0]. A fterwa rds, it is possibly wise to try some exhaust i ve minimis ation method for th ese automata [15]. Then, th e automat a also ha v e to be s ymbolic ally encoded [31, 26]. Furthermo re, the time spent on opt imising the automata has to be balanced aga inst th e overa ll compu tation ti me in order to a v oid runn ing out of time in the automaton optimisa tion step. 2 All in all, these aspe cts make the whole synthe sis process quit e complica ted and ar guab ly , less compe lling than other approa ches, which ulti- mately reduc es the publishabil ity of an y result on such a wo rkflo w , which in turn leads to little ince nti ve to perform researc h or write tools in this area. 5 Pr oviding a framework f or futur e e valu ations The prece ding sections discussed the dif ficulties of co mposing meaningfu l exp erimental ev aluations of synthe sis tools. Nev erthel ess, as benc hmarking is often conside red to be the only way to distinguis h promisin g ideas from the ones that are likely not to be useful (see, e.g., [71]), in this sectio n, we pro- pose thr ee evalu ation schemes for each of the proble ms of using appr opriate be nch marks and dealin g with comple x workflows whose co mpositio ns respect the dif ficulties discussed earlier . The schemes are ordere d from the minimum requirement to show th at a ne w techniq ue is worthwhile consid ering to the “super ior” sc heme that demonstr ates clear advan tages over pre vious techniques. 5.1 B enchmarking 5.1.1 Comparison using the home field advanta ge It is fa ir to say that a new appr oach should beat older app roache s at least in the cases in whic h it has a natura l adv antage. This is typically sh o wn by tak ing some example specification that f alls into the class of syste ms the ne w appro ach is inten ded to be applied to, applying a protot ype implementatio n of the approa ch to it, and sho wing that pre vious tools perf orm wo rse u sing an automatic, ingen uine, con version to the semantics/s copes of the prev ious tools. This means in particula r to con vert between Mealy- and Moore-ty pe semantics if applic able. Competitor tools which can o nly handl e a subset of the langu age of the ne w prototy pe tool need not be consider ed. 5.1.2 Comparison fr om a neutral view-point One problem of bench marking tools with differ ent scopes and semantics agains t the same examples is that speci ficatio ns ar e typical ly gea r ed towar ds the usage with a cert ain tool . A typi cal example is the pre-sy nthesi s pr ocess di scusse d in Sect ion 4 .2 t hat e nsures tha t t he sp ecificatio n of a system f alls in to th e class handled by the GR(1) s ynthe sis tool s. After th is has b een done, the sp ecificatio n is not only suitable b ut also optimised for such a tool. As many signallin g bits are introduced in the process, tools like L I L Y and A C AC I A 20 09 that are explicit in the input and output bit valuat ions ha v e proble ms with handling 2 In benchmark comparisons, it is customary to restrict the running t imes of the tools. Such a time restri ction is the typical answer to the problem that in most experimental ev aluations, there are some benchmark/tool combination s that do not yield a result ev en after days or weeks of computation time. R. Ehlers 11 such pre-sy nthes ized specificat ions eve n in cases in which they can deal with the non-pre-sy nthesiz ed ver sions. A similar situa tion arises for e xample when loca lising the assumptio ns (as discuss ed in Section 4.2): doing so is beneficial f or A C AC I A 2010, b ut render s the optimisati ons of U N B E A S T un usable as the inpu t is then no longer in the ( V assumpti ons ) → ( V guaran tees ) form. As a solution to this prob lem, w e propose the followin g sc heme: g i ven a setting, the spe cification is written for all tools to be compare d indi vidually , taking care of their specialiti es. If the prototype tool of a new ap proach perfo rms better in such a situation than pre vious tools, it is clear that th e techn iques propo sed ha ve their merits if used correctly when modelling a s pecificat ion. It shoul d be n oted, ho weve r , that this scheme fa v ours tools that re quire some form of pre-synthesi s: by re writing the specificati on for the simpler tool in a smart way , its performance can often greatly be increased. As an example , we refer to the work on re writing the AMB A AHB b us arbiter specification [30]. 5.1.3 Beating the other tool s w her e they hav e a natural adva ntage As a third scheme, we propo se that if a prototype implementatio n of some approach can beat other tools on benchmark suites on which they ha ve a natural adv antag e, this should suf fice to show the merits of a ne w approach without doubt. In order to do so, one would typically us e an automatic co n verter between the scope and semantic s (if necessary) of the other tool and the scope and semantics of the new protot ype tool to import benchmarks orig inally written for the other tool. The co n verter must not app ly sophis ticated optimisatio ns on the speci fication. As an examp le, con vert ing the L TL formula G FX p to GF p for some at omic proposi tion p dur ing the adaptat ion of th e Mealy/Moore -type semantics should be consid ered to be fai r , whereas re writin g a gu arantee int o a simpler on e tha t is only equi v alen t if th e gi ven assumpti ons al so hold is proba bly too complex for this scheme. 5.2 Comp lex workflo ws 5.2.1 Basic scheme In order to combat th e problem of ha vin g workflo ws that in v olv e multiple steps tha t can be be skip ped without obstructin g the steps follo wing (like for example automaton optimisation s), we propose the fol- lo wing scheme: for succe ssi v ely incr easing t imeout valu es (using a reasona ble gran ularity ), the synthesis approa ch is performed using the giv en timeout v alue for all indivi dual sub-step s in volv ed until the re- specti ve tool e xec ution yield s an ans wer . If the least timeout v alue that leads to a resul t for the n e w techni que is lo wer than the least such timeout valu e for pre viou s approaches, it is sho wn that the new approa ch has some m erits. 5.2.2 Advanced scheme As a n e xtensi on to the b asic scheme, it is worth while to sho w that the timeout value ob tained in the bas ic scheme does not make th e old approaches look bad unn ecessa rily . Let A be the least timeout v alue (for e ver y step of the workflo w) tried such that the prototype tool o f the ne w approac h termin ates w ith an answer . Let B be the ove rall running time of the process. If it can be shown that the other approaches do not ev en terminate with a timeo ut of B for each step, additional justificati on for the new approach is obtain ed. Of course, man y intermediat e v ariati ons betwee n the basic and a dv anced schemes are possib le. 12 Experiment al as pects of synthe sis 5.2.3 Simple scheme Probably th e most con vincing way to solv e the prob lem of hav ing compl ex workflo ws is to set stati c timeouts for the indiv idual steps of the workflo w and to just measure the overa ll runnin g time. If it is better than those of other tools, th is clearly sho ws the ef ficienc y of the ne w approach. Obviou sly , this scheme is hard to follo w when comparing ag ainst an o ther approach th at has itself man y steps which introd uce a need for indiv idual timeo uts if the auth or of the other tool has not provided good v alues for these. A dditio nally , special care must be taken not to “o ve rfit“ [21] the timeo uts for the indi vidual steps – tun ing the se v alues for th e ne w prototy pe tool aga inst a benchmark set a nd th en e valu ating on t he sa me set against the other tools in a publ ication is not fair and can be considered to be scientifical ly unso und. 6 Conclusion In this paper , w e discussed the pr oblems of e xper imentally ev alua ting a synthe sis tool. W e discus sed three major issues : the inc omparabl e semantics and scopes of the tools, the ba d comparability of the tools with respect to the benc hmarks a v ailabl e and th e compl exi ties of the workflo w s. Three e v alua- tion schemes to co mbat the first two of these problems and three sch emes to accoun t for th e comple x workflo ws ha ve been present ed. While the workflow ev alua tion sche mes cannot ful ly remo ve the prob lem that e xperi mental ev alua- tions using these are often not fully compellin g to the reader of a scientific publication , the y introd uce means o f comp aring to ols if they h a ve parts in their w orkflo w that may time out witho ut pr ohibit ing later steps (like, e.g., automaton optimisation). W e must admit that currently , ther e is no synthe sis tool that perfor ms such steps. H o we ver , this on its o wn is an interesting f act: due to the immense set of te ch- niques pro posed in the literat ure for reduc ing the sizes an d numbers o f automata repres enting the ov erall specifica tion, operation s such as dete rmining whether a guara ntee is actua lly necessa ry in a specifica tion or more complicated automato n minimisatio n techniqu es are not used in current tools yet ev en thoug h the theory behin d these operation s has been establishe d [33, 15, 18, 12]. Thus, we hope that the three workflo w ev alua tion schemes proposed help to lev el the way to fur ther adv ances in this area. As a final n ote, we would like to defend the argu mentatio n in this paper agains t the point of vie w tha t establ ishing a common file format with its clearly defined semanti cs and scope and requ iring all future tools to use it as a basis is a way to fight th e benchmark ing problem. W e ha v e sho wn in Secti on 4 that the choice of techn iques affec ts the choice of the semantics of a tool. Thus, picking one particula r sc ope and s emantics wou ld dri ve the ev olu tion of the tools an d thus also the th eory into a cert ain directio n while ignoring possibiliti es apart from technique s not suitable for the scope an d semantics agreed upon. Ho wev er , as e ven if leavin g this consideratio n apart, the form of a ”typical ” specification in practi ce (conju nction of guarante es [23, 45, 51] vs. ( V assumpti ons ) → ( V guaran tees ) for m [7, 16, 8, 5, 30]) is not agreed upon, it is fair to ar gue that consensu s will not be reached within the next fe w years. Also, an implicit or e xplic it requ irement th at tools with a complex workflo w should always be ev alua ted in a way similar to the simple scheme proposed here is highly problematic: due to the lo w number of meaning ful benchmark s, fixing good values as timeouts fo r the intermedi ate steps withou t ov erfitting for the co ncrete set of benchmarks in the e valuat ion is hardly po ssible . As a result, such a req uirement would basically rule out complex optimis ations a-priori (or require cheating by an author in the paper by overfitti ng), which is highly qu estion able for practica l approac hes to a problem that is, after al l, stil l 2EXPTIME-complete. R. Ehlers 13 Ackno wledgements This work was supported by the German Research Fou ndatio n (DFG) as part of the T ransre gion al Col- labora ti ve Research Center “ A utomatic V erification and Analysis of Complex Systems” (SF B/TR 14 A V A CS ). The autho r wants to thank Barbara Jobstmann and Emmanuel F iliot fo r helpf ul comments on the descrip tions of the techniqu es emplo yed in A N Z U , L I L Y an d A C A C I A . Refer ences [1] Mart´ ın Abadi, Leslie Lam port & Pierre W olper (1989) : Realizable and Un r ealizable Specification s of Reac- tive Systems . In Ausiello et al. [3], pp. 1–17, doi: 10 .1007/BFb0035748 . [2] ARM Ltd . (1999): AMBA TM specification (r ev . 2) . A vailable at www.arm.com . [3] Giorgio Ausiello, Mariangio la Dezani- Ciancaglini & Simona Ronch i Della Rocca, editors (1989) : Automata, Languages and Pr ogramming, 16 th Internation al Colloquium, (ICALP) . LNCS 372, Springer . [4] Arm in Biere, Marijn Heule, Hans van Maaren & T oby W alsh, editors (200 9): Hand book of Satisfiability . IOS Press. [5] Roder ick Bloem , K rishnendu Chatterjee, Karin Greimel, Thomas A. Hen zinger & Barbara Jobstman n (2010 ): Robustness in the Pres ence of Liven ess . In T ou ili et al. [72], pp. 41 0–424 , doi: 10.1007/ 978- 3- 642- 1429 5- 6_ 36 . [6] Roder ick Bloem, Alessandro Cimatti, Ka rin Greimel, Georg Ho fferek, Robert K ¨ onigho fer , Ma rco Roveri, V iktor Schu ppan & Richar d Seeber (2010) : RA TSY - A New Requir ements Ana lysis T oo l with Synthesis . I n T ouili et al. [72], pp. 42 5–429 , doi: 10.1007/978- 3- 642- 14295 - 6_ 37 . [7] Roder ick Bloem, Stef an Galler , Barbara Jobstmann, Nir Piterman, Amir P nueli & Martin W eiglhof er (2007): Interactive pr esentation: Automatic har dwar e synthesis fr o m specifications: a case study . In Rudy Lauwere- ins & Jan Madsen, editors: D A TE . A CM, pp. 118 8–119 3, do i: 10.1145/1266366 . 1 266622 . [8] Roder ick Bloem, Stef an Galler , Barbara Jobstmann, Nir Piterman, Amir P nueli & Martin W eiglhof er (2007): Specify , Compile, Run: Har dwar e fr om PSL . Electr . Notes Theor . Comput. Sci. 190(4 ), pp. 3–16 , do i: 10. 1016/j.entcs.2007.09.004 . [9] Roder ick Bloem, Barbara Jobstmann & Ma rtin W eiglhof er (2007 ): An zu . http: // www.ist.tugra z. at/ staff/jobs tmann/ anzu/ . [10] Rand al E. Bryant (19 86): Gr aph-Ba sed Algorithms for Boo lean Functio n Manipulation . IEEE Trans. Com- puters 35(8) , pp. 677– 691. [11] Fra nck Cassez, Alexandre Da vid, Emm anuel Fleury , Kim Guldstrand Larsen & Didier Lim e (2005 ): Effi- cient On-th e-Fly Algorithms for the An alysis of T imed Ga mes . In Mart´ ın Ab adi & Luca de Alfaro, editors: CONCUR . LNCS 3653, Springe r , pp. 66–80 , doi: 10.1007/11539 452_ 9 . [12] Lo renzo Clemente & Richard May r (2010) : Multip ebble Simulations for Alternating Automata - (Extend ed Abstract) . I n Paul Gastin & Franc ¸ ois Laroussinie, editors: CONCUR . LNCS 6269, Spr inger, pp. 297 –312, doi: 10.1007/978- 3- 642- 153 75- 4_ 21 . [13] La urent Doyen, Emmanuel Filiot, Naiyong Jin & Jean-Fra nc ¸ois Raskin (20 09): Acacia - L TL Rea lizability Check and W innin g Str ate gy Synthesis using Antichains . http: // www.anti chains . be/ac acia/ . [14] Alexand re Duret-Lu tz & Denis Poitrenaud ( 2004) : SPO T : An Extensible Model Ch ecking Lib rary Using T ransition-Based Generalized B ¨ uchi Automata . In Doug DeGroot, Peter G. Harriso n, Har ry A. G. W ijshoff & Zary Segall, editors: MASCO TS . IEEE Computer Society , pp. 76–83. [15] R ¨ udiger E hlers (2010 ): Minimising D eterministic B ¨ uchi Automata Pr e cisely Using S A T Solving . In Strichman & Szeider [70], pp. 326–33 2, doi: 10. 1007/978- 3- 642- 14186 - 7_ 2 8 . 14 Experiment al as pects of synthe sis [16] R ¨ udiger Ehlers (2010): Symbolic Bounded Synthesis . In T ou ili e t al. [ 72], p p. 365–37 9, doi: 10.1007/ 978- 3- 642- 1429 5- 6_ 33 . [17] R ¨ udiger Eh lers (20 10): Unbeast – Symb olic Bound ed S ynthesis . ht tp://react.cs.uni- saarlan d. de / tools/unbe ast/ . [18] R ¨ udiger Eh lers & Bernd Fink beiner (2010 ): On th e V irtue of P a tience: Minimizing B ¨ uchi Automata . In van de Pol & W eber [6 2], pp. 129–145 , d oi: 10.1007/978- 3- 642- 1616 4- 3_ 10 . [19] Cind y Eisner & Dana Fisman ( 2006) : A Practical Intr oduction to PS L (Series on I nte grated Cir cuits a nd Systems) . Springer-V erlag . [20] E. Allen Emerson & A. Prasad Sistla, editors (2000) : Computer Aided V erification , 12th Intern ational Con- fer ence, CA V 2000, Chicago, IL, USA, J uly 1 5-19, 2000, Pr oceeding s . LNCS 1855, Springe r . [21] Em anuel Falkenauer (1998): On Method Overfitting . J. Heuristics 4(3), pp. 281–2 87. [22] Em manuel Filiot, Naiyon g Jin & Jean -Franc ¸ ois Raskin (20 09): An Antichain A lgorithm for LTL Realizabil- ity . I n Ahme d Bou ajjani & Oded Maler, editors: CA V . LNCS 5643, Spring er , pp. 263– 277, d oi: 10.1007/ 978- 3- 642- 0265 8- 4_ 22 . [23] Em manuel Filiot, Naiyong Jin & Jean-Franc ¸ ois Raskin (2010): Compositional Algorithms for L TL Synthesis . In Ahme d Bouajjani & W ei-Ng an Chin, editors: A TV A . LNCS 6252, Springer, pp. 112–1 27, doi: 10.1007/ 978- 3- 642- 1564 3- 4_ 10 . [24] Bern d Finkb einer & Sv en Schewe ( 2005) : Uniform Distrib uted Synthesis . In : LICS . I EEE Computer Society , pp. 321– 330, doi: 10.1109/ LICS.2005.53 . [25] Mich ael J. Fisch er & Richar d E. Lad ner (197 9): Pr opositional Dynamic Logic of Re gular P r ograms . J. Comput. Syst. Sci. 18(2) , pp. 194– 211. [26] Riccar do Forth & Paul Molitor (2 000): An efficient heuristic for state encodin g minimizing the BDD repr e- sentations of the transistion r elation s of finite state machines . In: ASP-D A C . A CM, pp. 61–66. [27] Paul Ga stin & D enis Od doux (20 01): F ast LTL to B ¨ uchi Automata T ranslation . In G ´ erar d Berry , Hub ert Comon & Alain Finkel, editors: CA V . LNCS 2102, Springe r , pp. 53–65. [28] Allen V an Gelder & Daniel Le Berre, editor s ( 2010): Pragmatics o f SA T . W ork shop at th e Federated Logic Conferenc e (FLoC) 2010, Edinburgh. [29] Nag hmeh Ghafari, Alan J. Hu & Zvonimir Rakam aric (20 10): Context-Bounded T ranslations for Con- curr ent Softwar e: An Empirical E valuation . In v an de Po l & W eber [62], pp. 227–2 44, doi: 10.1007/ 978- 3- 642- 1616 4- 3_ 17 . [30] Y ashd eep Godhal, Krishn endu Chatterjee & Thoma s A . Henzin ger (2010): Synthesis of AMBA AHB fr om F ormal Spec ification . CoRR abs/1001 .2811. A vailable at http:/ / arxiv.org/abs/1001.2811 . [31] Wilsi n Gosti, Tiziano V illa, Alexander Saldanh a & Alberto L. Sangiovanni-V incentelli (2 007): FSM Encod- ing for BDD R epr esentations . Applied Mathematics and Compu ter Science 17(1) , p p. 113– 124, doi: 10. 2478/v1000 6- 007- 0011- 6 . [32] Er ich Gr ¨ ad el, W olfg ang Thomas & Thomas W ilke, edito rs (2002): Automata, Logics, and Infinite Games: A Guide to Curr e nt Researc h . LNCS 2500, Springer . [33] Kar in Greimel, Roderick Bloem, Barbara Job stmann & Mo she Y . V ard i (2008 ): Open Imp lication . In Luca Aceto , Ivan D amg ˚ ard, Leslie Ann Goldbe rg, Magn ´ us M. Halld ´ orsson, Anna Ing ´ olfsd ´ ottir & Igor W alu kiewicz, edito rs: ICALP (2 ) . LNCS 5126, Springer , pp. 3 61–37 2, do i: 10.1007/ 978- 3- 540- 7058 3- 3_ 30 . [34] Aid an Harding , Mark Ryan & Pierre-Yves Schobben s (2 005): A New Algorithm for Strate gy Synthesis in L TL Games . In Nicolas Halbwachs & Lenore D. Zuck, editors: T ACAS . LNCS 3440, Springer , p p. 477–49 2. [35] Th omas A. Henzinger & Nir Piterman (2 006): S olving Games W ithout Determinization . In Zo lt ´ an ´ Esik, editor: CSL . LNCS 4207, Springe r , pp. 395–410 , doi: 10.1007/11874683 _ 26 . [36] I BM Researc h: RuleBa se formal verification too l tutorial . https://www.research.ibm.com/haifa/ projec ts/ verifi cation / RB_Hom epage/ . R. Ehlers 15 [37] Barb ara Jo bstmann & Roderick Bloem (20 06): Lily - a LInear Lo gic sYnthesizer . http://www.iaik. tugraz . at/conte nt/research/ design _ verifi cation / lil y/ . [38] Barb ara Jo bstmann & Roderick Bloem (2006 ): Optimizations for LTL Synth esis . In: FMCAD . IEEE Com- puter Society , pp. 117–1 24, doi: 10.1109/FMCAD .2006. 22 . [39] Barb ara Jobstmann, Stefan Galler , Martin W eiglh ofer & Roder ick Bloem (200 7): Anzu : A T oo l for Pr o perty Synthesis . In W er ner Damm & Holg er Herm anns, ed itors: CA V . LNCS 4590, Springer, pp. 2 58–26 2, doi: 10 . 1007/978- 3- 540- 733 68- 3_ 29 . [40] Joa chim Klein & Christel Baier ( 2006) : Experiments with deterministic ω -automa ta for formulas of lin ear temporal lo gic . Theor . Comput. Sci. 363(2 ), pp. 182–1 95, doi: 10. 1016/j.tcs.2006.07.022 . [41] Ur i Klein & Amir Pnueli (2010) : R evisiting Synthesis of GR(1) Specifications . In : HVC . LNCS 6504. [42] Step han K ottler (2010) : S A T Solvin g with Refer en ce P oints . In Strichman & Szeider [70], pp. 1 43–15 7, doi: 10.1007/978- 3- 642- 141 86- 7_ 13 . [43] Jame s H. Kuku la & Thomas R. Shiple (2000 ): Buildin g Cir cuits fr om Relatio ns . In Emer son & Sistla [20], pp. 113– 123. [44] Or na Kupferm an (2006): A vo iding Determinization . In: LICS . IEEE Computer Society , pp. 2 43–25 4, doi: 10.1109/LICS.2006.15 . [45] Or na Kupf erman, Nir Piterman & M oshe Y . V ardi (200 6): Safraless Compositional Synthesis . In Thomas Ball & Robert B. Jones, editors: CA V . LNCS 4144, Springe r , pp. 31–44 , doi: 10.1007/11817 963_ 6 . [46] Or na Kupferm an & Moshe Y . V ardi (1 999): Churc h’ s p r ob lem re visited . Bulletin of Symbo lic Logic 5(2), pp. 245– 263. A vailable at http ://www. math.ucla.edu/ ~ asl/bsl/0502/0502- 00 4. ps . [47] Or na Kupferm an & Moshe Y . V ardi (2 001): Model Ch ecking of S afety Pr operties . Formal Metho ds in System Design 19(3) , pp. 291– 314. [48] Or na Kupferman & Moshe Y . V ardi (200 1): Synthesizing Distributed Systems . I n: LICS . [49] Or na Kupferma n & Moshe Y . V ardi ( 2005) : Safraless Decision Pr ocedur es . In: FOCS . IEEE, pp. 5 31–54 2, doi: 10.1109/SFCS.2005.66 . [50] Xin xin Liu & Scott A. Smo lka (19 98): S imple Linear-T ime Algorithms fo r Min imal F ixed P oints (Extended Abstract) . I n Kim G. Larsen, Sven Skyum & Glynn W inskel, editors: ICALP . LNCS 1443, Sprin ger , pp. 53–66 , doi: 10.1007/978- 3- 540- 647 81- 2_ 53 . [51] A. Morgenstern (2010 ): Symbo lic Contr oller Synthesis for LTL Specificatio ns . Ph. D. thesis, Depar tment of Computer Science, University of Kaiserslautern, Germany . [52] A. Morgenstern & K. Sch neider (2010 ): E xploiting the T e mporal Logic Hierar chy and the Non-Con fluence Pr op erty for Efficient LTL S ynthesis . In A. Montana ri, M. Napoli & M. Parente, editors: GandALF . EPTCS 25, pp. 89–10 2, doi: 10. 4204/EPTCS.25.11 . [53] Silvia M. M ¨ uller & W olfga ng J. P aul (2000 ): Computer ar chitectur e: co mplexity and corr ectness . Springe r . [54] Alexand er Nadel & V a dim Ryvchin (201 0): Assignme nt Stack Sh rinking . In Strich man & Szeider [70], pp. 375–3 81, doi: 10.1007/978- 3- 642- 141 86- 7_ 35 . [55] Carsten Sinz ( organiser): SA T -Ra ce 2010 . htt p://baldur.iti.uka.de/sat- race- 2010 / . [56] I an Parberry (1 994): A Guide for New Refer ees in Theor etical Comp uter Science . Inf. Comp ut. 112(1 ), pp. 96–11 6. [57] Nir Piterman, Amir Pnueli & Y aniv Sa’ar (200 6): Synthesis of Re active(1) Designs . In E. Allen Emerson & Kedar S. Namjoshi, editor s: VMCAI . LNCS 3855, Springer, pp . 364–38 0, doi: 10.1007/116097 73_ 24 . [58] Am ir Pnueli (1977) : Th e T empo ral Lo gic of Pr ograms . In : FOCS . IE EE, pp. 46–57. [59] Am ir Pnueli & Roni Rosner (1989 ): On the Synthesis of a Reactive Module . I n: POPL . pp. 179–1 90. [60] Am ir Pnueli & Roni Rosner (198 9): On the Synthesis of an Async hr o nous Re active Module . In Ausiello et al. [3], pp. 652– 671. 16 Experiment al as pects of synthe sis [61] Am ir Pnu eli, Y an i v Sa’ar & Leno re D. Zuck (20 10): Jtlv: A F ramework for Developing V erifica tion Algo- rithms . In T ou ili et al. [72], pp. 171–174, doi: 10.1007/978- 3 - 642- 14295- 6_ 18 . [62] Jaco van de Pol & Michael W eb er , editors (2010): Model Chec king So ftwar e - 17th Intern ational S PIN W o rkshop, Enschede, Th e Netherland s, Se ptember 27-2 9, 20 10 . LNCS 6349, Springer, doi: 10.1007/ 978- 3- 642- 1616 4- 3 . [63] Kr istin Y . Rozier & Moshe Y . V ard i (2010 ): LT L satisfiability checking . STTT 12(2) , pp. 123 –137, do i: 10. 1007/s1000 9- 010- 0140- 3 . [64] Sh muel Safra (1 989): Complexity of Automata o n I nfinite Objects . Ph.D. thesis, W eizman n I nstitute of Science, Rehovot, Israel. [65] Sven Schewe & Bernd Fink beiner (200 7): Bound ed Synthesis . I n Kedar S. Namjoshi, T omohir o Y oned a, T e ruo Higashin o & Y oshio Okamu ra, ed itors: A TV A . LNCS 4762, Spring er , pp. 4 74–48 8, doi: 10 .1007/ 978- 3- 540- 7559 6- 8_ 33 . [66] Saq ib Sohail & Fabio Somenzi (2009 ): S afety first: A two-stage algorithm for LTL games . In: FMCAD . IEEE, pp. 77–8 4, doi: 10.1109/FMCAD . 2009.535113 8 . [67] Saq ib Sohail, Fabio Somen zi & Kavita Ravi ( 2008): A Hybrid Algorithm for LTL Games . In Fran cesco Logozzo , Do ron Peled & Lenore D. Zu ck, editors: VMCAI . LNCS 4905, Sprin ger , pp. 3 09–32 3, doi: 10. 1007/978- 3- 540- 781 63- 9_ 26 . [68] Fabio Somen zi (2009): CUDD: CU Decision Diagram P ackage Release 2.4.2 . [69] Fabio Som enzi & Rod erick Bloem (2000 ): Efficient B ¨ uchi A utomata fr om LTL F ormulae . In Emerso n & Sistla [20], pp. 248– 263. [70] Of er Strichman & Stefan Szeider, editors (2 010): Theory and App lications of Sa tisfiability T esting (SAT) . LNCS 6175, Springe r , doi: 10.1007/978- 3- 642- 14186 - 7 . [71] W alter F . Tichy (1998): Shou ld Computer Scientists Experiment Mor e? IEEE Computer 31(5) , pp. 32–4 0. [72] T ayssir T ou ili, Byron Cook & Paul Jack son, editors (20 10): The 22nd International Confer ence on C omputer Aided V erification (CA V) . LNCS 6174, Springer, doi: 10.1007/978- 3 - 642- 14295- 6 . [73] Mo she Y . V ardi (19 95): An Automata-Theoretic Appr oach to Linear T empo ral Logic . In Faron Moller & Graham M. Birtwistle, editors: Banff Higher Order W or kshop . LNCS 1043, Springe r . [74] Mo she Y . V ardi & L arry J. Stockmeyer (1985) : Impr oved Upper and Lo wer Bou nds for Mod al Logics of Pr ograms: Pr elimina ry Report . In: STOC . A CM, pp. 240–251 .