Formalizing and validating properties in Asmeta with Large Language Models (Extended Abstract)

F ormalizing and v alidating prop erties in Asmeta with Large Language Mo dels (Extended Abstract) Andrea Bom barda 1 [0000 − 0003 − 4244 − 9319] , Silvia Bonfan ti 1 [0000 − 0001 − 9679 − 4551] , Angelo Gargan tini 1 [0000 − 0002 − 4035 − 0131] , and Nico P ellegrinelli 1 [0009 − 0000 − 4944 − 6845] Univ ersity of Bergamo, Bergamo, Ita ly {andrea.bombarda, silvia.bonfanti, angelo.gargantini, nico.pellegrinelli}@unibg.it Abstract. W riting temp oral logic properties is often a challenging task for users of model-based developmen t frameworks, particularly when translating informal requirements in to formal sp ecifications. In this pa- p er, w e explore the idea of integrating Large Language Models (LLMs) in to the Asmeta framework to assist users during the definition, formal- ization, explanation, and v alidation of temporal prop erties. W e present a w orkflow in which an LLM-based agent supports these activities by lev eraging the Asmeta sp ecification and the feedback produced b y the mo del chec k er. Thi s work serves as a pro of of concept that illustrates the feasibility and p oten tial b enefits of suc h an integration through rep- resen tative examples. Keyw ords: Asmeta , T emporal Logic, Large Language Mo dels 1 In tro duction Mo dern soft w are and system engineering pro jects are increasingly complex, often in v olving heterogeneous comp onen ts, tight safety constraints, and short develop- men t cycles. T raditional dev elopmen t approaches, which rely hea vily on man ual co ding and late-stage testing, struggle to ensure early v alidation, traceability , and consistency b et w een design and implementation. This leads to higher costs and risks, esp ecially in safet y-critical domains suc h as automotive, aerospace, and healthcare. Model-Based Developmen t (MBD) offers an effectiv e alterna- tiv e. By relying on executable mo dels, it enables early simulation and v erification of system b ehavior, reduces am biguit y in requirements through formal or semi- formal representations, supp orts automatic code generation to ensure alignment b et w een design and implemen tation, and enhances traceability and compliance with industry standards. A v ariet y of to ols supp ort MBD, and, in this work, we fo cus on the Asmeta framew ork [1,3]. The framework allo ws dev elopers to work with Abstract State Machines (ASMs) [4], which extend Finite State Machines (FSMs) b y replacing unstructured control states with states that can con tain arbitrarily complex data. 2 A. Bom barda et al. 2 Ob jectiv es The Asmeta framework pro vides a suite of to ols that supp ort developers through- out the soft w are lifecycle, including design, dev elopmen t, and operation. In this w ork, we concen trate on the design phase, and in particular on verification activi- ties, which are carried out using the mo del chec k er in tegrated into AsmetaSMV [2]. Starting from the functional requirements, Asmeta enables users to mo del a system using the AsmetaL language. Users can embed Computation T ree Logic (CTL) and Linear T emp oral Logic (L TL) properties directly within the ASM mo del, expressing them o v er the Asmeta signature. The AsmetaSMV to ol, then, automatically translates the ASM mo del (including b oth the system mo del and the L TL/CTL prop erties) into a mo del of the symbolic mo del c hec k er NuSMV [6], whic h is used for v erification. Based on feedback collected from Asmeta users, particularly students enrolled in courses where Asmeta is taught, we observed that many of them encoun ter dif- ficulties when writing L TL and CTL prop erties. While formulating prop erties in natural language is generally straightforw ard, c hallenges arise when translating these descriptions in to the corresp onding AsmetaL expressions. T o address this issue, we prop ose integrating Large Language Mo dels (LLMs) in to the workflo w. W e hav e identified, after analyzing existing works, at least four w a ys in which LLMs can assist the designer in the v erification of TL properties. O1 . Assisting during the definition of the pr op erty in natur al language: Given an Asmeta specification, LLMs can b e leveraged to interpret the domain con- text and automatically deriv e properties, expressed in natural language, that are v alid within that context. These prop erties are neither intended to b e ex- haustiv e nor universally guaranteed; rather, they serve to improv e the user’s understanding of the domain and to p oten tially reveal constraints or assump- tions that were implicit in the original requiremen ts. There are already some exp erimen ts prop osing Retriev al-Augmen ted Property Generation in other con- texts, lik e Prop ert yGPT [7]. O2 . Helping the designer to write the c orr esp onding L TL/CTL pr op erties: Given an Asmeta specification that includes the system b eha vior and prop erties ex- pressed in natural language, the LLM automatically translates these prop erties in to their corresp onding L TL or CTL formulations within the Asmeta speci- fication. By doing so, we aim to reduce the gap betw een informal requirement descriptions and formal verification artifacts, ultimately improving usability and lo w ering the en try barrier for new users. F urthermore, the generated prop erties must b e formulated with aw areness of, and consistency with, the functions defined in the Asmeta sp ecification, thereb y ensuring that the resulting L TL/CTL formulas are consistent with the op era- tional semantics of the mo del. This implies that the LLM must b e context-a ware: it must recognize the roles and types of functions, and incorp orate these elements appropriately in to temp oral operators. Title S uppressed Due to Excessive Length 3 O3 . Explanation of the TL pr op erties using natur al language: F ormalizing re- quiremen ts enables systematic reasoning ab out inconsistencies, the detection of am biguities, and the identification of critical issues in system mo dels. T emporal logic formulae are a natural choice for sp ecifying requiremen ts related to desired system b eha viors. How ever, understanding and effectively using temporal logic demands a strong formal bac kground. Therefore, there is a need for approac hes that mak e temporal logic form ulae more interpretable for engineers, domain ex- p erts, and other stakeholders in v olv ed in the developmen t pro cess. F or this goal, w e assume that LLMs can be used to translate TL prop erties back to natural language. F or this scope, we plan to adopt (or adapt, if necessary) the technique presen ted in [5]. O4 . V alidation of the TL pr op erties using witnesses and c ounter examples: When a property fails v erification, AsmetaSMV pro duces a coun terexample that can b e automatically exp orted as an Avalla scenario. Lik ewise, successful executions could b e exp orted as scenarios that highligh t witnesses. Large Language Mo dels (LLMs) can supp ort dev elop ers’ understanding by translating Avalla scenar- ios into natural language, thereby clarifying their meaning and describing the b eha vior they implement. W e rep ort on preliminary in v estigations fo cused on outlining the pro cess to b e adopted for in tegrating an LLM-based assistant in to the Asmeta framework. The goal at this stage is to define the pro cess required to incorp orate an LLM- based assistan t in to the Asmeta framew ork. 3 Metho dology and W alkthrough Demonstration This w ork follo ws a multi-step metho dology aimed at designing, integrating, and empirically ev aluating an LLM-based assistan t for the automatic genera- tion of temp oral prop erties in Asmeta sp ecifications. Since this is a preliminary study , our fo cus is on defining the ov erall pro cess, iden tifying key challenges, and conducting initial feasibility analyses rather than delivering a fully op erational framew ork. The LLM-based assistan t, whose workflo w is rep orted in Figure 1, should b e able to generate and explain L TL and CTL properties that are not only syn- tactically correct, but also meaningful with respect to the given Asmeta mo del. This implies that the assistan t m ust interpret the signature of the sp ecification, understand the types of functions, and generate form ulas or explanations that consisten tly refer to elemen ts defined b y the mo del. The user begins b y providing b oth an Asmeta specification and the infor- mal requirements of prop erties to b e formalized, within a sp ecifically designed prompt. The assistant, then, extracts relev ant signatures from the Asmeta sp eci- fication and translates the requirements into an L TL or CTL form ula em b edded in to the Asmeta sp ecification, or pro vides explanations complying with the re- quested tasks and using the same functions as in the Asmeta sp ecification. 4 A. Bom barda et al. LLM-Based Ag ent AsmetaSMV Asmeta sp ecification Prompt Enric hed Asmeta sp ecification Fig. 1. Our solution w orkflow. asm Clo ck import StandardLibra ry signature : domain Second subsetof Integer domain Minute subsetof Integer domain Hour subsetof Integer monitored signal: Bo olean controlled sec: Second controlled min: Minute controlled h: Hour definitions : domain Second = {0 : 59} domain Minute = {0 : 59} domain Hour = {0 : 23} macro rule r_IncMinHours = par if min = 59 then h := (h + 1) mo d 24 endif min := (min + 1) mo d 60 endpar main rule r_Main = if signal then par if sec = 59 then r_IncMinHours[] endif sec := (sec + 1) mo d 60 endpar endif default init s0: function sec = 0 function min = 0 function h = 0 Listing 1. Asmeta sp ecification of a simple Clo c k W e exp ect users to write requirements b y following the EARS (Easy Ap- proac h to Requirements Syn tax) [8], a structured w a y to write requirements so they are clear, consisten t, and less am biguous. In the follo wing, w e describ e ho w we en vision that our approach w ould help in eac h of the four ob jectives describ ed in Section 2. Our demonstration is based on direct interactions with the ChatGPT 5.2 Instant mo del and aims at chec king the feasibilit y of our approac h. Let us consider the Asmeta sp ecification of a simple clock as in Listing 1. F or the ob jective O1 , users may ask the LLM-Based Agent to list the thr e e most imp ortan t prop erties guaranteed by the Asmeta sp ecification. In this case, the LLM replies with the three follo wing prop erties (each with its o wn description): – Time variables always stay within valid clock ranges; – Correct carry-over from seconds to minutes and hours; – Time advances only when the signal is true. Although some of these properties may be straightforw ard, providing a nat- ural language explanation can help lo wer the barrier to understanding Asmeta sp ecifications. F or ob jective O2 , users ma y ask the LLM-Based Agent to encode in CTL a prop ert y verifying that when the min function reaches the v alue 59 , it is set to 0 in the next state. With this prompt, the LLM-Based Agen t replies with the follo wing CTL prop ert y: Title S uppressed Due to Excessive Length 5 scenario ClockScenario load Clo ck.asm set signal := true; step ; check sec = 1 and min = 0 and h = 0; set signal := false; step ; check sec = 1 and min = 0 and h = 0; Listing 2. Avalla scenario for the Clock sp ecification AG (min = 59 implies AX (min = 0)) A prop ert y such as the one generated by our LLM-Based Agent must contain only v alid op erators and functions. While LLMs can b e fed with an Asmeta sp ec- ification, it may happ en that the LLM generates a syn tactically wrong prop erty . F or this reason, as shown in Figure 1, we integrate our solution with AsmetaSMV . Once the property is generated, the mo del c hec k er’s feedbac k is leveraged. If errors are detected, the mo del c heck er’s output is pro vided together with the prompt to correct the prop ert y formulation. F or ob jectiv e O3 , suppose that the considered Asmeta sp ecification contains the property given b efore. Users may leverage our LLM-Based Agent to explain in natural language what the prop erty means . In this example, the output of our Agen t is the follo wing: In every reachable state of the system, if the value of min is 59, then in all possible next states the value of min will be 0. Finally , for the ob jective O4 , if the Avalla scenario in Listing 2 is av ailable (e.g., as the output of the counterexample generation), users may b e interested in ha ving its high-lev el to b etter understand the reasons behind a p ossible failure. In this case, the LLM-Based Agen t pro vides the follo wing explanation: The scenario verifies that the clock advances by one second when the signal is true, and stops advancing when the signal is false, preserving the current time. 4 Conclusion In this pap er, w e explored the idea of integrating Large Language Mo dels into the Asmeta framework to supp ort users in writing, understanding, and v alidating temp oral logic prop erties. Motiv ated by the difficulties encountered b y practi- tioners when translating informal requirements in to L TL and CTL sp ecifications, w e outlined a conceptual w orkflow in whic h an LLM-based agen t assists the de- signer across multiple v erification-related tasks, ranging from prop ert y elicitation in natural language to the in terpretation of witnesses and coun terexamples. Through simple examples, we show ed that LLMs can plausibly bridge the gap b et ween informal requirements and formal verification artifacts, while remaining consisten t with the semantics of Asmeta sp ecifications and the feedback pro vided b y the mo del chec ker. F uture work will fo cus on turning this conceptual workflo w in to a concrete implemen tation within Asmeta . 6 A. Bom barda et al. References 1. Arcaini, P ., Bombarda, A., Bonfanti, S., Gargantini, A., Riccob ene, E., Scan- durra, P .: The ASMET A approach to safet y assurance of soft ware systems. In: Logic, Computation and Rigorous Metho ds, pp. 215–238. Springer (2021). https: //doi.org/10.1007/978- 3- 030- 76020- 5_13 2. Arcaini, P ., Gargantini, A., Riccob ene, E.: AsmetaSMV: A W ay to Link High-Lev el ASM Mo dels to Low-Lev el NuSMV Sp ecifications, p. 61–74. Springer Berlin Hei- delb erg (2010). https://doi.org/10.1007/978- 3- 642- 11811- 1_6 3. Bom barda, A., Bonfanti, S., Gargantini, A., Riccob ene, E., Scandurra, P .: ASMET A T o ol Set for Rigorous System Design, p. 492–517. Springer Nature Switzerland (Sep 2024). https://doi.org/10.1007/978- 3- 031- 71177- 0_28 4. Börger, E., Stärk, R.: Abstract State Machines. Springer Berlin Heidelb erg (2003). https://doi.org/10.1007/978- 3- 642- 18216- 7 5. Cherukuri, H., F errari, A., Sp oletini, P .: T o wards explainable formal metho ds: F rom L TL to natural language with neural machine translation. In: Gerv asi, V., V ogelsang, A. (eds.) Requirements Engineering: F oundation for Softw are Quality . pp. 79–86. Springer In ternational Publishing, Cham (2022) 6. Cimatti, A., Clark e, E., Giunc higlia, E., Giunc higlia, F., Pistore, M., Ro v eri, M., Sebastiani, R., T acc hella, A.: NuSMV 2: An Op enSource T ool for Sym b olic Mo del Chec king, p. 359–364. Springer Berlin Heidelb erg (2002). https://doi.org/10. 1007/3- 540- 45657- 0_29 7. Liu, Y., Xue, Y., W u, D., Sun, Y., Li, Y., Shi, M., Liu, Y.: Propertygpt: Llm- driv en formal verification of smart contracts through retriev al-augmented property generation. In: Pro ceedings 2025 Net work and Distributed System Security Sym- p osium. NDSS 2025, In ternet Society (2025). https://doi.org/10.14722/ndss. 2025.241357 8. Ma vin, A., Wilkinson, P ., Harwoo d, A., Nov ak, M.: Easy approac h to requirements syn tax (ears). In: 2009 17th IEEE International Requirements Engineering Confer- ence. p p. 317–322 (2009). https://doi.org/10.1109/RE.2009.9