Machine learning and AI research for Patient Benefit: 20 Critical Questions on Transparency, Replicability, Ethics and Effectiveness

Machine learning (ML), artificial intelligence (AI) and other modern statistical methods are providing new opportunities to operationalize previously untapped and rapidly growing sources of data for patient benefit. Whilst there is a lot of promising research currently being undertaken, the literature as a whole lacks: transparency; clear reporting to facilitate replicability; exploration for potential ethical concerns; and, clear demonstrations of effectiveness. There are many reasons for why these issues exist, but one of the most important that we provide a preliminary solution for here is the current lack of ML/AI- specific best practice guidance. Although there is no consensus on what best practice looks in this field, we believe that interdisciplinary groups pursuing research and impact projects in the ML/AI for health domain would benefit from answering a series of questions based on the important issues that exist when undertaking work of this nature. Here we present 20 questions that span the entire project life cycle, from inception, data analysis, and model evaluation, to implementation, as a means to facilitate project planning and post-hoc (structured) independent evaluation. By beginning to answer these questions in different settings, we can start to understand what constitutes a good answer, and we expect that the resulting discussion will be central to developing an international consensus framework for transparent, replicable, ethical and effective research in artificial intelligence (AI-TREE) for health.

💡 Research Summary

The paper addresses a critical gap in the rapidly expanding field of machine learning and artificial intelligence (ML/AI) applied to health care: the lack of systematic guidance ensuring that research is transparent, reproducible, ethical, and demonstrably effective (the “TREE” criteria). While many promising studies exist, most fail to meet these standards, risking wasted effort, potential harm, and slow translation into clinical practice. To remedy this, the authors propose a structured framework consisting of 20 concrete questions that span the entire lifecycle of an ML/AI health project—from inception through data analysis, model evaluation, impact assessment, and implementation.

The first, overarching question asks how the model is embedded within feedback loops of a learning health system, emphasizing that AI should not be a static “lone wolf” but part of an iterative cycle where data, decisions, and outcomes continuously inform each other. In the inception phase, three questions focus on defining a clear clinical problem that delivers patient benefit, determining when and how patients should be involved throughout data collection, analysis, deployment, and use, and ensuring organizational transparency about data flow. These address data sovereignty, consent, and public trust.

The analysis phase contains eight questions that scrutinize data suitability (capturing real‑world heterogeneity and quality), methodological realism (reflecting constraints of data collection and storage), data accessibility for other researchers, adequacy of computational resources and software, relevance of reported performance metrics to the intended clinical context, clinical justification of any statistical performance gains, comparison against current best technologies and appropriate baselines, and full reproducibility of the modeling pipeline (including code, parameters, random seeds). An additional question probes external validity—whether results hold in settings beyond the development environment.

Impact evaluation includes three questions assessing whether the algorithm exacerbates inequities across protected characteristics, whether clinicians and patients find the model interpretable and trustworthy, and whether real‑world effectiveness in the target clinical setting is demonstrated with solid evidence.

Finally, the implementation phase asks whether the model will be regularly re‑assessed and updated as data quality and clinical practice evolve, whether it is cost‑effective to build, deploy, and maintain, how financial benefits will be distributed if commercialized, and how regulatory approval requirements have been satisfied.

These 20 questions were derived from a collaborative dialogue among a broad spectrum of UK and international stakeholders—including the Alan Turing Institute, HDR UK, NICE, MHRA, CPRD, EQUATOR, Stanford’s METRICS, and the University of Chicago’s DSSG—reflecting diverse perspectives from regulators, academia, and industry. By systematically answering these questions across multiple projects, the community can begin to define what constitutes a “good” answer, iteratively refine the checklist, and ultimately converge on an international consensus framework (AI‑TREE) for transparent, reproducible, ethical, and effective AI research in health. The authors argue that adopting such a framework, together with emerging policies like the NHS AI Code of Conduct, will build trust, prevent the propagation of ineffective or harmful algorithms, and ensure that AI innovations truly improve patient outcomes.