Rejoinder: Expert Elicitation for Reliable System Design

Statistic al Scienc e 2006, V ol. 21, No. 4, 460– 462 DOI: 10.1214 /0883423 06000000556 Main article DO I: 10.1214/0883 42306000000510 c  Institute of Mathematical Statisti cs , 2006 Rejoinder: Exp ert Elicitation fo r Reliable System Design Tim Bedfo rd, John Quigley and L esley W alls First of all, w e w ould like to thank the discussan ts for the care and though tfulness that they ha v e tak en in preparing their commen ts. Ko ehler p resen ts a helpful discussion, putting for- w ard a num b er of different ideas that generaliz e the approac h tak en. A taxonom y for tec hnical system elicitat ion would pro vide useful guidance f or practi- tioners and s er ve to co dif y applicable assump tions during the different systems engineering p hases. Al- though more researc h is needed h ere, one could see the emergence of in ternational sta nd ard s that rely on suc h a taxonom y . W e ac knowle dge that the elicit ation pr oblem v aries greatly dep ending on the tec hnical system as p oin ted out b y Ko e hler and w e ha ve sough t to generalize our exp erience in studyin g complex systems, includ- ing aerospace, rail and na v al for b o th commercial and defense mark ets. This explains our bias to ward the “closed loop” case. W e agree with the t wo ex- tra areas of exp ert elicitation iden tified for “wa- terfall” cases: lac k of exp ertise cont inuit y and the problem of “forw ard casting” requirement s for an existing system. Both of these relate to discon tinu- ous change s in system op eration. Suc h changes ha v e o ccurred most ob viously in m ilitary systems and other pro jects with long lead times. Ho wev er, in the commercial wo rld, such discon tin uities can b e forced by regulatory or market c h anges, or b y out- sourcing decisions. These ma y mak e h istoric data collect ion taxonomies less relev an t to th e reliabilit y questions p osed to supp ort new op erational deci- sions and, therefore, provide new areas of applica- tion for exp ert jud gement tec hniques. The final p oi nt r aised b y Ko ehler ab out the diffi- culties imp osed by system complexit y is w ell mad e This is an electro nic reprint of the original article published by the Institute of Mathematical Statistics in Statistic al Scienc e , 2006, V ol. 2 1, No. 4, 460–4 62 . This reprint differs from the original in pag ination a nd t yp ogr aphic deta il. and the notion of multiple concurrent reliabilit y mo d- els is in triguing. Th is do es partially link int o the n o- tion of exp ert weigh ting. Ho wev er, it also requires a go o d unders tanding of the notion of mo del “exp er- tise” as d istinct fr om exp ert “exp ertise.” On e migh t argue that if sufficien t un derstanding exists to b e able to quanti fy mo d el exp ertise, then one should b e able to directly build a meta mo del that in corp o- rates the b e st of eac h mo del. In practice, the need to b e cost-efficien t will usually m itigate aga inst such a strategy , and mo del com bination is an int eresting alternativ e. W ang r igh tly observes that w e ha ve n ot tried to giv e a survey of exp ert ju dgemen t metho dologies. The m ain reason for this is that sev eral surve ys ha ve b een undertaken, in cluding a recen t one with a wide co verag e (Jenkinson, 2005 ). It has not b ee n our pur- p ose to su rve y th ese metho ds again. Instead we aim to discuss th e con text in wh ic h suc h mo d els ma y b e used in the engineering design pr o cess and to sho w that the exp ert pr oblem in this context frequently is more demanding than a “straigh tforw ard” proba- bilit y elicitatio n. Ha vin g said this, W ang is right to id en tify em- pirical Ba yes (EB) as an in teresting metho d with p oten tial application in th e area under discuss ion. There is, how eve r, more than one wa y to utilize this approac h. T he appr oac h discussed b y W ang explic- itly uses exp ert information as data , h en ce forcing the analyst to c ho ose priors and like liho o d s for the exp ert d ata giv en the p arameters. This is a fun da- men tal problem b e cause it forces the analyst into the r ole of meta exp ert. In this case, the sp ecifica- tion of p ( x | Θ) is going to b e p roblematic whether or not w e use EB. In our o wn w ork with EB (Quigley , Bedford and W alls, 2006 , 2007 ) w e ha v e integ rated exp ert jud gemen t in to the app roac h through the selection of p ools that comprise differen t t yp es of ev en ts whose d ata are merged in the EB p ro cess. The use of EB allo ws us to increase the quanti t y of data a v ailable to mak e estimates of reliabilit y p a- rameters through exp ert j udgemen ts ab out w hic h 1 2 T. BEDFORD, J. QUIGLEY A N D L. W AL LS ev en ts should h a v e similar ord er of magnitude b e- ha vior. W ang’s prop osal for using eviden tial reasoning in reliabilit y com bines a num b er of differen t question- able features. F or the purp oses of this r ejoinder, w e prop ose distinguishing three d ifferen t issues con- tained in the discussion: • Nonprobabilistic representat ions of uncertain t y . • Imprecise uncertain ties. • Multicriteria decision mo d els. Nonpr ob abilistic r epr esentations of unc e rtainty : W e are y et to b e con vinced that these play a useful role. The examples we h av e seen discussed—b oth exam- ples to sho w th e limitations of pr obabilit y and exam- ples to sho w the n eed for a more general framew ork— are marred b y lac k of clarit y ab out the underly- ing problem b eing mo d eled. Ind eed th is sometimes seems to b e the p oin t of the “need” for something else. I n m an y cases m ore atten tion paid to struc- turing the p roblem and articulating the reasons for mo deling will surely tak e care of man y of the am b i- guities. T o paraph rase O’Hagan and Oakley ( 2004 ), who r ecen tly wr ote a pap er titled “Probabilit y is p erfect, but w e can’t elicit it p erfectly ,” we might sa y that “probabilit y is p erfect, b ut we fin d it difficult to apply approp r iately .” Such d ifficulties are ev en more apparen t wh en applied to more complex generaliza- tions of pr ob ab ility . The danger is that theoretici ans use suc h metho ds as a fix to a v oid r esolving imp or- tan t mo deling issu es. Impr e cise unc e rtainties : Th ere is gro w ing in ter- est, and some sound foun dational w ork, in the area of interv al probabilities. Suc h quan tities ma y ha v e a r eal and useful application, p articularly in b oun d- ing probabilities of undesirable ev ents. See, for ex- ample, Coolen, Co olen-Sc h rijner and Y an ( 2002 ), Co olen and Y an ( 2003 ), Co olen ( 2004 , 2006 ), Au- gustin and Co olen ( 2004 ) and Co olen and Co olen- Sc hrijner ( 2005 ). Multicriteria de c ision mo dels : It is imp ortant not to confuse such mo d els, w h ic h in the fir st instance are designed to represent trade-offs b et we en differ- en t attributes of a decision consequence, w ith proba- bilistic mo dels that represent system and knowledge relationships. In the case of the motorcycle men- tioned in the discu s sion, the motorcycle is mo deled most simp ly as a series system in the subsystems men tioned. T he discus s ion of this example seems to force th e analyst d o wn a more complex r oute th at ig- nores the b asic engineering stru cture of the system. F u rthermore, so many elemen ts of the calculation app ear to b e arb itrary—for example, w hat is the ev en t “that the i th basic attribute supp orts the h y- p othesis that the general attribute is assessed to the n th grade” that is b eing ascrib ed a probabilit y and wh y should w eigh ts from Saat y’s analytic hierarch y pro cess b e used to m ultiply pr obabilities?—that it is d ifficult to see that this leads to something r e- ally meaningful and of m ore use than other simpler rule-of-th umb ev aluations. The exp er ience of F ento n and Neil in dev elop- ing Ba ye sian metho d s, esp ecially Bay esian n et works, adds v aluable supp ort to man y issues raised in the pap er. W e would certainly ac knowle dge that TRACS is an early example of a meta mo deling system of the t yp e we discuss and it is go o d to h ear that mo del building in its more recent d ev elopmen ts is faster. Unfortunately , b ecause these are commercial sys- tems, it is difficult for academics to b e able to mak e judgement s ab ou t the internal w orkings of the sys- tems. W e agree with the p oint raised by F en ton and Neil that the customer can b e an exp ert, as w ell as clien t, b ecause it will often b e the case that th e customer p ossesses exp ertise ab out, for example, the op era- tional en vironm en t and maintenance of the family of s y s tems. Hence the b oundaries b et w een the man- ufacturer and customer classes in T able 1 should b e take n as an example of t ypical stak eholder roles rather than as a fixed allocation appr opriate for all systems. In those cases where the customer h as dual roles, additional care is required to manage bias that arises due to the lev els of trust. Our limited exp eri- ence to date in working with teams that sp an stak e- holder classes has b een m ixed : w e ha v e exp erienced a lac k of op enness in some situations, w hile in others w e enjo ye d a sharing in b oth directions motiv ated by the need for a useful decision s upp ort tool. The pres- ence of trust will b e influen ced by the culture of the companies in v olv ed as well as the exp ected longevit y of the r elationship. T he a wareness and managemen t of sub jectiv e bias is imp ortan t, but we agree that it should not b e r egarded as a reason not to conduct Ba yesia n mo deling. In the absence of muc h relev ant empirical data, F enton and Neil p oint out that reliabilit y assessmen t can b e regarded a “blac k art.” Certainly , Ba ye sian mo deling can help to mak e assump tions more trans- paren t. Ho wev er, to some exten t this simply brin gs with it a shift of d ifficult y from one area of mo del- ing to another. T he parties hav e to find some lev el REJOINDER 3 of agreemen t on pr ior distribu tions, whic h can b e problematic if the p arties really understand the sig- nificance of the c hoice b eing made. F enton and Neil giv e examples of the use o f ex- p ert elicita tion within six-sigma ap p roac h es. This is notew orthy giv en that man y reliabilit y pr ob lems arise from systematic d esign v ariation d u e to man- agemen t as we ll as tec hnical considerations. Despite the strong relationship b et we en reliabilit y and qual- it y , culturally they can b e disparate within organi- zations. By f o cusing on failure mo d e ident ification and trac king, we ha v e exp erienced limited success in conceptually reeliciting pr iors for reliabilit y mo d- eling using p ro du ction exp erience (W alls, Quigley and Marshall, 2006 ). The reasons for only limited success can b e partially attributed to common pr o- cess drivers iden tified b y th e aerospace companies in v olv ed in mo deling. F or example, the difficulties of using standard data-driven statistical pro cess con- trol for lo w-vo lume man ufacturing has facilita ted rather than hindered the ac ceptance of elicitation. Ho wev er, we emphasize that th e conceptual accep- tance by stak eholders as evidence of success in us e currentl y remains scarce. Hence the resea rch ques- tions p osed concerning cultural conflict, organiza- tional d riv ers and pro cess drive rs are imp ortan t to address iss u es for whic h only piecemeal anecdotal evidence currently exists. W e w ould lik e to clarify to F enton and Neil that w e are not assuming implicitly or otherwise that the b enefits of probabilit y elicitation only accrue in sit- uations where there is already a highly develo p ed reliabilit y methodology and w e do agree th at elici - tation pla ys a distinctiv e role in organizations where it is not cost-effectiv e to collect empirical data. Ho w- ev er, in situations where a highly d evelo p ed reliabil- it y culture exists, there is a critical need to structure the mo dels b eing quant ified, an d the users w ill cer- tainly ben efi t from that stru ctur ing phase, as well as the later quan tification. F enton and Neil p oin t out that the “additional k ey b enefi t” of this kind of probability elicitation in terms of pro viding co dified information for fu- ture systems is one that is certainly of imp ortance in those industries with very short d ev elopmen t cy- cles. F or systems with longer cycles, there is time to collect op erational in formation to up d ate or replace the exp ert deriv ed data, and industry “generic data bases” pla y the role discussed. W e are grateful to the discussants for their com- men ts, w hic h p r o vide fur th er insigh ts in to many is- sues raised in the pap er and con tribute a num b er of new ideas that w ere not explored within the original pap er. REFERENCES Augustin, T. an d Coolen, F. P. A. (2004). Nonparamet- ric predictive inference and interv al p robabilit y . J. Statist. Plann. Infer enc e 124 251–272. MR2080364 Coolen, F. P. A . (2004). On the use of imprecise proba- bilities in reliabilit y . Quality and R eliabil ity Engine ering International 20 193–202. Coolen, F. P. A. (2006). On nonparametric p redictive in - ference and ob jectiv e Ba yesianis m. J. L o gi c , L anguage and Information 15 21–47. MR2254566 Coolen, F. P. A. and Coolen-Schrijne r, P. (2005). Non- parametric predictive reliabilit y demonstration for failure- free p eriod s. IMA J. M anag. M ath. 16 1–11. MR2124165 Coolen, F. P. A ., Coolen-Schrijne r, P. and Y an, K. J. (2002). Nonp arametric p redictive inference in reliabilit y . R eli ability Engine ering and Syste m Safety 78 185–19 3. Coolen, F. P . A. and Y an, K. J. (2003). Nonparametric predictive inference for grouped lifetime data. Re liabil ity Engine ering and System Safety 80 243–252. Jenkinson, D. (2005). The elicitation of probabilities—a re- view of th e statistical literature. BEEP wo rking pap er, Univ. Sheffield. O’Hagan, A. and Oakley, J. E. (2004 ). Probabilit y is p er- fect, bu t we can’t elici t it p erfectly . R eliabil ity Engine ering and System Safety 85 1–3, 239–248. Quigley, J., B edford, T. and W alls, L. (2006). F ault tree inference for one-shot devices using Ba yes and empirical Ba yes metho ds. In Pr o c. ESREL 2006 Saf ety and R eliabil - ity C onfer enc e 859–865. Quigley, J., B edford, T. and W a lls, L. (2007). Estimat- ing rate of occurrence of rare even ts with empirical Ba yes: A railw a y application. R eliability Engine ering and System Safety 92 . T o app ear. W alls, L., Qu igley, J. and Marshall, J. (2006). Mo d eling to supp ort reliabilit y enhancement during pro duct devel- opment with applications in the UK aerospace industry . IEEE T r ans. Engine ering Management 53 263–27 4.