Selection and Collider Restriction Bias Due to Predictor Availability in Prognostic Models

Selection and Collider Restr iction Bias Due to Predictor A v ailability in Prognostic Models Marc Delord 1,* 1 School of Lif e Course & P opulation Sciences, Depar tment of P opulation Health Sciences, King’ s College London, London, United Kingdom * Correspondence: marc.delord@kcl.ac.uk, T el: +44 20 7848 8710 In medicine, prognostic scores or clinical prediction models are statistical models intended to estimate an individual’ s probability of e xperiencing a specific health outcome ov er a defined period, based on their clinical and non-clinical characteristics [ 1 – 4 ]. Outcomes can refer to clinical e vents such as disease onset, complications , referr al to secondar y care, organ f ailure, or death. Classical e xamples of prognostic scores include the Fr amingham cardiov ascular risk score, or iginally dev eloped to estimate long-ter m coronar y hear t disease r isk, and more recent tools such as the QRISK3 score, widely used in UK primar y care to predict 10-year cardiov ascular risk. The inflation in the number of pub lished prediction models over recent decades has gener ated an e xtensive methodological liter ature on common shor tcomings in prognostic model de velop- ment and guidelines for their design, validation, and repor ting [ 2 , 4 – 11 ]. In broad ter ms, the de velopment of prediction models should include the dev elopment of the prediction model itself [ 6 ], its external validation and calibration [ 1 , 7 ], and, ideally , the conduct of a prospectiv e clinical and economic impact study [8]. This well-defined fr amework, intended to ensure the methodological soundness of prediction model assessment implicitly assumes that required predictors are routinely a v ailable at the point of care [ 2 , 6 ], an assumption that, despite its impor tance, has receiv ed little e xplicit attention in the methodological literature on prediction models [ 11 ]. More generally , when a prognostic score is dev eloped or v alidated using retrospective data, inclusion is restr icted–by construction– to patients with recorded predictors. This restr iction to patients with recorded predictors is not inherently problematic , and if measurement occurs independently of determinants of outcome risk, the analysed sample ma y still yield unbiased estimates . Howe v er , when predictor measurement depends on under lying disease se verity or related care processes, restriction selects individuals based on a v ar iable influenced by determinants of the outcome . This process is analogous to protopathic bias, a f or m of re vers causality in which early manif estations of disease prompt inter vention bef ore formal diagnosis [12]. This situation, illustrated in panel A of Figure 1, corresponds to classical selection bias: underly- ing disease se verity (U) influences both the outcome and the likelihood of predictor measurement 1 so that restricting analysis to individuals with recorded predictors eff ectively selects on se v er ity . U Y I { P 1 i s m e a s u r e d } U Y I { P 2 i s m e a su r e d } P 1 U Y I { P 2 i s m e a s u r e d } P 1 A B C Figure 1: Selection and collider restriction bias ar ising from conditioning on predictor a vailability in prognostic models . Directed acyclic graphs are used to illustrate causal dependencies between obser v ed and unobser v ed v ar iables . U represents unobser ved disease sev er ity; P 1 and P 2 repre- sent predictors; and Y represents the outcome. Fr amed nodes indicate measurement-based restriction. P anel A depicts simple selection bias, where underlying disease sev er ity triggers measurement of the predictor . P anel B depicts simple selection bias in which disease se verity is captured through a measured pro xy . P anel C illustrates collider restriction bias, where both underlying disease se verity and its pro xy influence measure- ment of P 2 , making its av ailability a collider . A concrete e xample illustrating ho w the assumption of unbiased predictor a vailability ma y not hold in practice is provided b y the Kidney F ailure Risk Equation (KFRE) [ 15 ]. The KFRE is a prognostic model dev eloped to predict progression to kidney f ailure in patients with chronic kidney disease stages 3–5 (CKD 3–5), defined by persistent reduction in kidney function or markers of kidney damage . It estimates an individual’ s risk of kidney f ailure at 2 or 5 years and is intended to inform risk str atification and referr al decisions in patients with chronic kidne y disease. The commonly used f our-variable version relies on age, sex, estimated glomerular filtration rate (eGFR), and urine albumin-to-creatinine ratio (uA CR), a measure of albuminuria. Although the KFRE has been the subject of numerous v alidation studies repor ting strong pre- dictiv e performance [ 16 – 19 ], its uptake in routine clinical practice remains limited [ 19 ]. This limited use ma y be par tly e xplained by constr aints in routine data a vailability [ 20 ], with eGFR or uA CR not being systematically recorded in community-based care f or patients with chronic kidney disease stages 3-5. In the UK, albuminuria testing among patients with chronic kidne y disease stages 3–5 remains uncommon in primar y care, with f ewer than 25% undergoing uA CR testing within one y ear ov erall, b ut increasing to about 37% among those f or mally registered 2 with chronic kidney disease, indicating substantially higher testing conditional on chronic kidne y disease recognition [ 21 ]. More recent national audits repor t annual testing in around 30% of patients with chronic kidne y disease stages 3–5 [ 22 ]. Similar patter ns hav e been repor ted in the US, where albumin uria testing remains uncommon among adults at r isk f or chronic kidney disease, with A CR recorded in around 17% of these patients , while being associated with a higher pre valence of chronic kidney disease treatment [ 23 ]. More generally , a recent systematic re view and meta-analysis of 59 studies across 24 countries, including ov er 3 million patients with chronic kidne y disease, sho wed that while 81.3% of patients receiv ed eGFR monitoring, only 47.4% underwent alb uminuria testing [24]. The example of the KFRE suggests alter native scenar ios, illustrated in panels B and C of Figure 1. P anel B ref ers to a situation in which a set of predictors displa y diff erent patter ns of missingness: a measured predictor ( P 1 ) ser ves as a pro xy f or unobser ved underlying disease se verity . As predictor P 1 deteriorates, fur ther clinical inv estigation is under taken, leading to measurement of predictor P 2 , thereb y restr icting computation of the prognostic score to patients with recorded P 2 . As in panel A, this situation leads to classical selection bias. Adding a direct causal relation between underlying disease sev erity and the measurement of P 2 —as shown in panel C—makes the a v ailability of P 2 a collider between P 1 and underlying disease sev erity . When the situation reduces to classical selection bias, prognostic model de velopment ma y still yield coefficients representative of the underlying higher-risk population. By contrast, condition- ing on a collider distor ts associations between all baseline predictors—not only P 1 and P 2 —and the outcome [13]. In the conte xt of the KFRE, declining eGFR prompts uACR testing, while perceived ov erall kidney f ailure r isk—reflected by symptoms and comorbidities such as diabetes—independently influences the same decision. This double dependence of predictor av ailability on both eGFR and the perceiv ed r isk of the outcome characterises collider restr iction bias [25]. Bey ond consequences regarding the applicability of prognostic models [ 11 ], patter ned av ailability of predictors raises broader methodological issues in model de velopment and v alidation, and suppor ts consideration of simplified prognostic models [ 26 – 28 ] in the setting of prospectiv e cohor ts [ 2 ]. T aken together , these considerations highlight predictor a vailability as a central, yet often implicit, assumption underpinning the dev elopment, validation, and use of prognostic models. 3 References [1] Douglas G Altman and Patric k Ro yston. What do we mean b y validating a prognostic model? Statistics in medicine , 19(4):453–473, 2000. [2] Karel G M Moons, P atr ick Ro yston, Yv onne V ergouwe , Diederick E Grobbee , and Dou- glas G Altman. Prognosis and prognostic research: what, wh y , and how? BMJ , 338:375, 2009. [3] Ewout W . Ste yerberg. Applications of prediction models. In Clinical Prediction Models: A Practical Approach to De v elopment, V alidation, and Updating , pages 11–31. Springer , New Y ork, 2009. [4] Ewout W Ste yerberg, Karel GM Moons, Danielle A v an der Windt, Jill A Ha yden, P ablo P erel, Sara Schroter , Richard D Rile y , Harr y Hemingwa y , Douglas G Altman, and Prog ress Group . Prognosis research strategy (progress) 3: prognostic model research. PLoS medicine , 10(2):e1001381, 2013. [5] Jeremy C W yatt and Douglas G Altman. Commentar y: Prognostic models: clinically useful or quickly f orgotten? Bmj , 311(7019):1539–1541, 1995. [6] P atrick Ro yston, Karel GM Moons, Douglas G Altman, and Yvonne V ergouwe . Prognosis and prognostic research: de veloping a prognostic model. Bmj , 338, 2009. [7] Douglas G. Altman, Yv onne V ergouwe, P atr ick Ro yston, and Karel G.M. Moons. Prognosis and prognostic research: v alidating a prognostic model. BMJ , 338:1432–1435, 5 2009. [8] Karel G M Moons, Douglas G Altman, Yv onne V ergouwe, and P atrick Ro yston. Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ , 338:606, 2009. [9] Harr y Hemingwa y , P eter Croft, P ablo P erel, Jill A Hayden, K eith Abrams, Adam Timmis , Andrew Briggs, Ruzan Udumy an, Karel GM Moons, Ew out W Ste yerberg, et al. Prognosis research strategy (progress) 1: A framework f or researching clinical outcomes. BMJ , 346, 2013. [10] GS Collins , JB Reitsma, DG Altman, and KGM Moons. T ransparent repor ting of a multiv ar i- able prediction mode l for individual prognosis or diagnosis (tripod): The tr ipod statement. British Jour nal of Cancer , 112(2):251, 2015. [11] Orestis Efthimiou, Michael Seo , K onstantina Chalk ou, Thomas Debra y , Matthias Egger , and Georgia Salanti. Dev eloping clinical prediction models: a step-by-step guide. BMJ , 386, 2024. [12] Alv an R. F einstein. Clinical Epidemiology: The Architecture of Clinical Research . W . B. Saunders, Philadelphia, 1985. [13] Sander Greenland. Quantifying biases in causal models: classical confounding vs collider- stratification bias . Epidemiology , 14(3):300–306, 2003. [14] Haidong Lu, Gregg S Gonsalves , and Daniel Westreich. Selection bias requires selection: the case of collider stratification bias . American jour nal of epidemiology , 193(3):407–409, 2024. [15] Navdeep T angri, Lesley A Ste vens , John Gr iffith, Hocine Tighiouar t, Ognjenka Djurdjev , David Naimark, Adeera Le vin, and Andrew S Le ve y . A predictive model f or prog ression of chronic kidney disease to kidne y f ailure. Jama , 305(15):1553–1559, 2011. 4 [16] Mieke J P eeters, Arjan D van Zuilen, Jan AJG van den Brand, Michiel L Bots, P eter J Blankestijn, J ack FM W etzels, MASTERPLAN Study Group , P eter J Blankestijn, Michiel L Bots, Marjolijn v an Buren, et al. V alidation of the kidney f ailure r isk equation in european ckd patients . Nephrology Dialysis T r ansplantation , 28(7):1773–1779, 2013. [17] Navdeep T angri, Morgan E Grams, Andrew S Le ve y , Josef Coresh, La wrence J Appel, Brad C Astor , Gabriel Chodick, Allan J Collins , Ognjenka Djurdjev , C Raina Elle y , et al. Multinational assessment of accur acy of equations f or predicting risk of kidne y f ailure: a meta-analysis. J ama , 315(2):164–174, 2016. [18] Ruper t W Major , David Shepherd, J ames F Medcalf, Gang Xu, Laur a J Gra y , and Nigel J Brunskill. The kidne y f ailure risk equation for prediction of end stage renal disease in uk primar y care: an external validation and clinical impact projection cohor t study . PLoS medicine , 16(11):e1002955, 2019. [19] Chav a L Ramspek, Ype de Jong, F r iedo W Dekk er , and Merel v an Diepen. T owards the best kidney failure prediction tool: a systematic revie w and selection aid. Nephrology Dialysis T ransplantation , 35(9):1527–1538, 2020. [20] Eugene J . Fine and M. Donald Blauf ox. Chapter 5: Referr al to specialists and models of care. Kidne y Inter national Supplements , 3:112–119, 1 2013. [21] Simon DS F raser , Julie P arkes, David Culliford, Mir iam Santer , and Paul J Roderick. Timeliness in chronic kidney disease and albumin ur ia identification: a retrospective cohor t study . BMC f amily practice , 16(1):18, 2015. [22] Dorothea Nitsch, Benjamin Caplin, Sally A. Hull, Da vid C . Wheeler , Lucy Kim, and Fiona Clear y . National chronic kidney disease audit: National repor t (par t 1), 2017. [23] Chi D Chu, F ang Xia, Y uxian Du, Rak esh Singh, Delphine S T uot, Julio A Lamprea- Montealegre , Ralph Gualtieri, Nic k Liao, Sheldon X K ong, T odd Williamson, et al. Estimated pre valence and testing f or alb uminuria in us adults at risk f or chronic kidne y disease . JAMA network open , 6(7):e2326230–e2326230, 2023. [24] Daniel Bekele K etema, Hannah W allace, W or kagegnehu Hailu, Sunil V Badve, P aul Ronksley , Brendon L Neuen, Rober to P ecoits-Filho , Mar tin Gallagher , Sradha K otwal, Vlado P er ko vic, et al. Quality of care for people with chronic kidne y disease: a systematic re view and meta-analysis . BMJ open , 15(9):e102044, 2025. [25] Stephen R Cole, Rober t W Platt, Enr ique F Schisterman, Haitao Chu, Daniel Westreich, David Richardson, and Charles P oole. Illustrating bias due to conditioning on a collider . Inter national jour nal of epidemiology , 39(2):417–420, 2010. [26] Jeremy C . Wy att and Douglas G. Altman. Prognostic models: clinically useful or quickly f orgotten? BMJ , 311(7019):1539–1541, 1995. [27] Ewout W . Ste yerberg, Marinus J . C. Eijk emans, F rank E. Harrell, and J . Dik F . Habbema. Simplifying a prognostic model: a simulation study based on clinical data. Statistics in Medicine , 23(16):2567–2586, 2004. [28] Michael C Sachs, Ar vid Sjölander , and Er in E Gabr iel. Aim for clinical utility , not just predictiv e accuracy . Epidemiology , 31(3):359–364, 2020. 5