Applying a Requirements-Focused Agile Management Approach for Machine Learning-Enabled Systems

Machine Learning (ML)-enabled systems challenge traditional Requirements Engineering (RE) and agile management due to data dependence, experimentation, and uncertain model behavior. Existing RE and agile practices remain poorly integrated and insufficiently tailored to these characteristics. This paper reports on the practical experience of applying RefineML, a requirements-focused approach for the continuous and agile refinement of ML-enabled systems, which integrates ML-tailored specification and agile management approaches with best practices derived from a systematic mapping study. The application context concerns an industry-academia collaboration project between PUC-Rio and EXA, a Brazilian cybersecurity company. For evaluation purposes, we applied questionnaires assessing RefineML’s suitability and overall acceptance and semi-structured interviews. We applied thematic analysis to the collected qualitative data. Regarding suitability and acceptance, the results of the questionnaires indicated high perceived usefulness and intention to use. Based on the interviews, stakeholders perceived RefineML as improving communication and facilitating early feasibility assessments, as well as enabling dual-track governance of ML and software work, allowing continuous refinement of the model while evolving the overall software project. However, some limitations remain, particularly related to difficulties in operationalizing ML concerns into agile requirements and in estimating ML effort.

💡 Research Summary

This paper addresses the growing mismatch between traditional Requirements Engineering (RE) and agile management practices and the distinctive characteristics of machine‑learning (ML)‑enabled systems, namely data dependence, experimental cycles, and model uncertainty. To bridge this gap, the authors propose RefineML, a requirements‑focused agile management approach that integrates three well‑established building blocks: (1) PerSpecML for multi‑perspective RE, (2) Agile4MLS for adapting Scrum to ML workflows, and (3) Lean R&D concepts such as early feasibility checks, Minimal Viable Model (MVM), and Layers of Done (LoD) for progressive model maturation.

RefineML is organized into three sequential phases inspired by Lean R&D: Initial Specification, parallel Conception and Technical Feasibility, and Agile Development. In the Initial Specification phase, the Product Owner together with domain experts, designers, data scientists, and engineers use PerSpecML to capture requirements across five perspectives (System Objectives, User Experience, Infrastructure, Model, Data). The resulting artefacts remain intentionally flexible to accommodate new insights. During Conception, two backlogs are created – a traditional software backlog (User Stories) and an ML backlog composed of Data Stories and Model Stories, as defined by Agile4MLS. This dual‑track backlog enables clear separation of ML‑intensive and non‑ML work while preserving traceability to the original PerSpecML concerns.

The Agile Development phase adopts Scrum cycles modified by Agile4MLS: the “two‑sprints‑ahead” principle ensures that the ML team works ahead of the software team, delivering a Demo API that mimics expected model outputs. This allows software developers to continue implementation while the ML team refines the model. Model evolution is governed by the MVM concept—an early, deployable model that delivers measurable business value—and by LoD, a set of maturity levels each defined by explicit evaluation metrics and acceptance criteria. LoD provides a transparent roadmap for model versioning and stakeholder communication.

The approach was evaluated in an industry‑academia collaboration between PUC‑Rio and EXA, a Brazilian cybersecurity firm. The joint project aimed to develop ML‑based fraud‑detection features. Over six months, three models progressed from initial MVMs to higher LoD levels. Evaluation combined a Technology Acceptance Model (TAM) questionnaire (measuring perceived usefulness, ease of use, and intention to use) with semi‑structured interviews of the business owner and the project lead. Quantitative results showed high perceived usefulness (average 4.3/5) and strong intention to continue using RefineML (average 4.1/5). Qualitative findings highlighted three major benefits: (a) improved communication and shared understanding among data scientists, developers, and business stakeholders; (b) early feasibility assessment through MVMs that reduced risk; and (c) dual‑track governance that kept ML and software work synchronized while allowing independent progress.

Nevertheless, participants reported persistent challenges. Translating high‑level ML concerns from PerSpecML into concrete Data/Model Stories proved labor‑intensive and sometimes ambiguous. Estimating effort for data‑centric activities (collection, labeling, preprocessing) remained unreliable, leading to sprint planning difficulties. Ethical considerations (fairness, transparency) were not fully integrated into the agile ceremonies.

The authors discuss these limitations and propose future improvements: tool support for automated mapping of PerSpecML concerns to agile backlog items; incorporating “uncertainty buffers” in sprint planning to accommodate estimation variance; embedding ethical checklists into sprint reviews; and extending RefineML to other domains such as healthcare and autonomous systems.

In conclusion, RefineML demonstrates that a requirements‑centric, continuously refined agile framework can effectively manage the complexities of ML‑enabled system development. It aligns early, stakeholder‑driven requirement capture with iterative model validation, thereby reducing risk and accelerating value delivery. The study provides empirical evidence of its suitability and acceptance, while also outlining concrete research directions to enhance scalability, automation, and ethical integration.