Application of Predictive Model Selection to Coupled Models

Application of Predictiv e Mo del Selection to Coupled Mo dels Gabriel T erejan u ∗ , T o dd Oliv er † , Chris Simmons ‡ Cen ter for Predictiv e Engineering and Computational Sciences (PECOS) Institute for Computational Engineering and Sciences (ICES) The Univ ersit y of T exas at Austin Abstr act — A predictive Ba yesian model selection approac h is presen ted to discriminate coupled mod- els used to predict an unobserved quantit y of inter- est (QoI). The need for accurate predictions arises in a v ariety of critical applications suc h as climate, aerospace and defense. A model problem is in tro- duced to study the prediction yielded b y the cou- pling of t wo ph ysics/sub-comp onen ts. F or each single ph ysics domain, a set of mo del classes and a set of sen- sor observ ations are av ailable. A goal-orien ted algo- rithm using a predictiv e approach to Bay esian mo del selection is then used to select the com bination of single physics mo dels that b est predict the QoI. It is shown that the b est coupled mo del for prediction is the one that pro vides the most robust predictive distribution for the QoI. Keywor ds: Pr e dictive Mo del Sele ction, Quantity of In- ter est, Mo del V alidation, De cision Making, Bayesian A nalysis 1 In tro duction With the exp onen tial gro wth of a v ailable computing p o w er and the contin ued developmen t of adv anced n u- merical algorithms, computational science has undergone a rev olution in which computer mo dels are used to simu- late increasingly complex phenomena. Additionally , such sim ulations are guiding critical decisions that affect our w elfare and securit y , such as climate c hange, performance of energy and defense systems and the biology of dis- eases. Reliable predictions of suc h complex physical sys- tems requires sophisticated mathematical mo dels of the ph ysical phenomena inv olved. But also required is a sys- tematic, comprehensive treatment of the calibration and v alidation of the mo dels, as well as the quantification of the uncertain ties inherent in such mo dels. While re- cen tly some atten tion has been paid to the propagation of uncertaint y , considerably less attention has b een paid to the v alidation of these complex, m ultiphysics mo dels. ∗ E-mail address: terejanu@ices.utexas.edu. Address: The Uni- versit y of T exas at Austin, 1 University Station, C0200 Austin, TX 78712, USA † E-mail address: oliver@ices.utexas.edu ‡ E-mail address: csim@ices.utexas.edu This becomes particularly c hallenging when the quantit y of interest (QoI) cannot be directly measured, and the comparison of model predictions with real data is not p os- sible. Suc h QoIs may be the catastrophic failure of the thermal protection system of a space shuttle reentering the atmosphere or the differen t p erformance c haracteris- tics of nuclear w eap ons in order to maintain the nuclear sto c kpile without undergoing underground n uclear test- ing. In this pap er, w e present an in tuitive interpretation of the predictiv e mo del selection in the context of Bay esian anal- ysis. While the predictive mo del selection is not an new idea, see Refs.[3, 4, 7], here we emphasize the connection b et w een the QoI-aw are evidence and the Ba y esian model a veraging used for estimation. This new interpretation of the Ba y esian predictive mo del selection rev eals that the b est mo del for prediction is the one which pro vides the most robust predictive probabilit y density function (p df ) for the QoI. Also, the latest adv ances in Marko v Chain Mon te Carlo (MCMC) [2] and estimators based on the k -nearest neighbor [8] are used to compute the information theoretic measures required in the problem of predictiv e selection of coupled mo dels. It is further argued that equiv alence b etw een predictiv e mo del selec- tion and conv en tional Ba y esian model selection can b e reac hed by p erforming optimal exp erimental design [6] for model discrimination. The structure of the paper is as follows: first the selection problem of coupled mo dels is stated in Section 2. The conv entional Bay esian mo del selection is describ ed in Section 3 and the extension to QoI-a ware evidence is deriv ed in Section 4. The mo del problem and numerical results are presented in Section 5 and Section 6 respectively . The conclusions and future w ork are discussed in Section 7. 2 Problem Statement Here w e are in terested in the prediction of a coupled mo del. The problem of selecting the b est coupled mo del in the con text of the QoI is to find the combination of sin- gle ph ysics mo dels that b est predict an unobserved QoI in some sense, see Fig. 1. Thus at the single physics lev el, we ha v e tw o physics A and B, eac h with a mo del class set, M A and M B , and a set of observ ations D A and D B resp ectiv ely . The cardinalit y of the tw o sets of mo del classes are |M A | = K A , |M B | = K B , and the the definition of mo del classes in eac h set is given by the state equations and measurement mo dels as follows, M A i : ( r A i ( u A i , θ A i ) = 0 y A = y A i ( u A i , θ A i ) M B j : ( r B j ( u B j , θ B j ) = 0 y B = y B j ( u B j , θ B j ) . CALIBR A TIONS PREDICTIONS M A 1 M A A K M A 2 physics A ... DA T A DA T A physics B M 1 B M 2 B M B K B ... coupling q decision making M A A K M 2 B Figure 1: Predictive Selection for Coupled Mo dels All the p ossible couplings of single physics mo dels yield the set of coupled mo del classes M = { M ij = ( M A i , M B j ) | M A i ∈ M A , M B j ∈ M B } with cardinality |M| = K A K B . The definition of a coupled mo del class in this set is given b y the state and measuremen t equation, and in addition w e also hav e the mo del for the QoI, M ij :      r ( u A i , u B j , θ A i , θ B j ) = 0 y AB = y AB ij ( u A i , u B j , θ A i , θ B j ) q = q AB ij ( u A i , u B j , θ A i , θ B j ) . Ha ving the set of all the coupled models, the selection problem becomes finding the b est coupled model in the set, M , for prediction purp oses. 3 Ba y esian Mo del Selection In the context of M -closed p ersp ectiv e, the conv entional Ba yesian approac h to mo del selection is to c ho ose the mo del which has the highest p osterior plausibility , M ∗ = arg max M π ( M | D , M ) . (1) Giv en the data at the single physics lev el, one can com- pute the p osterior model plausibility for all the models in the M A and M B sets, as the pro duct b et w een evidence and prior plausibility , π ( M A i | D A , M ) ∝ π ( D A | M A i , M ) π ( M A i |M ) . (2) The evidence is obtained during the calibration pro cess for each single-ph ysics mo dels, and it is giv en by the nor- malization constant in the Ba y es rule, used to compute the p osterior p df of mo del parameters, π ( θ A i | D A , M A i , M ) = π ( D A | θ A i , M A i , M ) π ( θ A i | M A i , M ) π ( D A | M A i , M ) . With no data at the coupled level, one can easily obtained the p osterior plausibilit y for coupled mo dels as, π ( M ij | D , M ) = π ( M A i | D A , M ) π ( M B j | D B , M ) . (3) Notice, that the b est coupled mo del is given by coupling the best mo dels at the s ingle physics level. Computa- tionally this is adv an tageous as there is no need to build all the p ossible coupled models using the single ph ysics mo dels. Lo oking at just t w o coupled mo dels, w e can say that w e prefer model M 1 o ver mo del M 2 if and only if π ( M 1 | D , M ) > π ( M 2 | D , M ). This inequality can be re- casted as the follo wing product of ratios: π ( D | M 1 ) π ( D | M 2 ) | {z } Bay es factor π ( M 1 , M ) π ( M 2 , M ) | {z } prior o dds > 1 (4) The log-evidence is the trade-off b et w een model complex- it y and how well the mo del fits the data. In other w ords the evidence yields the best mo del that ob eys the law of parsimon y , ln[ π ( D | M 1 )] = E [ln[ π ( D | θ , M 1 )]] − KL  π ( θ | D , M 1 ) || π ( θ | M 1 )  , (5) where the mo del complexity is giv en by the Kullbac k- Leibler divergence b et ween posterior p df and prior p df of mo del parameters [2]. Therefore, this mo del selection sc heme makes use of the follo wing information in c ho os- ing the best mo del: mo del complexit y , data fit, and prior kno wledge. Since it ob eys Occam’s razor, we implicitly gain some robustness with respect to predictions. How- ev er, if we hav e tw o differen t QoIs that we would like to predict, it is not obvious if the mo del selected under this sc heme will b e able to pro vide equally goo d predictions for b oth QoIs. This is due to the fact that the informa- tion about the QoI is not explicitly used in the selection criterion. In the follo wing sections, we will presen t an extension of mo del selection scheme to also account for the QoI, and discuss its implications. 4 Predictiv e Mo del Selection Giv en a mo del class set M = { M 1 , M 2 , . . . , M K } (for simplicit y the double index will b e ignored in the mo del class notation), and a set of observ ations D = { d 1 , d 2 , . . . , d n } generated by an unknown mo del from M , we are concerned with the problem of selecting the b est mo del to predict an unobserved quan tit y of in ter- est q . The selection of the mo del which b est estimates the p df of the QoI is seen here as a decision problem [7]. First one has to find the predictiv e distribution for each mo del class and the selection of the b est mo del class is based on the utilit y of its predictiv e distribution. Giv en all the av ailable information, the Bay esian predictiv e dis- tribution conditioned on a mo del M j is given by: π ( q | D , M j ) = Z π ( q | θ j , D , M j ) π ( θ j | D , M j ) d θ j (6) where the p osterior p df for mo del parameters is computed using Ba yes rule. In the followings it is assumed that the true distribution of the QoI is generated b y a mo del M j ( θ j ) ∈ M j , called the true mo del. Thus, the true p df of the QoI can b e written as: π ( q | θ , s , D, M ) = K X j =1 π ( q | θ j , D , M j ) s j (7) where θ = ( θ 1 , θ 2 , . . . , θ K ), s = ( s 1 , s 2 , . . . , s K ), and s j = 1 if and only if the true mo del b elongs to mo del class M j . Let U q ( θ , s , M j ) describ e the utility of choosing the p df asso ciated with mo del class M j as the predictive distribution of the QoI, q . Here the utility function is de- fined as the negativ e Kullback-Leibler div ergence of the true distribution and the predictive distribution of M j : U q ( θ , s , M j ) = − KL  π ( q | θ , s , D , M ) || π ( q | D, M j )  (8) The mo del that maximizes the follo wing exp ected utilit y (QoI-a ware evidence) is the mo del of c hoice for predictiv e purp oses: M ∗ = arg max M j ∈M Z U q ( θ , s , M j ) π ( θ , s | D, M ) d θ d s | {z } E θ , s [ U q ( θ , s ,M j )] (9) With few mathematical manipulations, the exp ected util- it y can b e written as follows, E θ , s [ U q ( θ , s , M j )] = Z U q ( θ , s , M j ) π ( θ | s , D, M ) π ( s | D , M ) d θ d s = K X i =1 π ( M i | D, M ) Z U q ( θ i , s i , M j ) π ( θ i | D, M i ) d θ i (10) = − K X i =1 π ( M i | D, M ) E θ i  KL  π ( q | θ i , D , M i ) || π ( q | D, M j )  4.1 In terpretation of the exp ected utilit y used in mo del selection Consider now that only t wo mo del classes exist in our mo del set M = { M 1 , M 2 } . W e prefer mo del class M 1 o ver M 2 and write M 1  M 2 if and only if: E θ , s [ U q ( θ , s , M 1 )] > E θ , s [ U q ( θ , s , M 2 )] (11) Substituting Eq.(10) into Eq.(11) the following model se- lection criterion can b e derived: R ( M 1 || M 2 ) R ( M 2 || M 1 ) | {z } Risk ratio π ( D | M 1 ) π ( D | M 2 ) | {z } Bay es factor π ( M 1 , M ) π ( M 2 , M ) | {z } Prior o dds > 1 (12) where the n umerator in the predictive risk ratio is given b y the following expressions. The denominator is ob- tained by analogy with the numerator. R ( M 1 || M 2 ) = E θ 1  KL  π ( q | θ 1 , D , M 1 ) || π ( q | D, M 2 )  − E θ 1  KL  π ( q | θ 1 , D , M 1 ) || π ( q | D, M 1 )  (13) The mo del selection criterion in Eq.(12) can b e inter- preted as the evidence of model class M 1 in fa vor of mo del class M 2 , and is comp osed of prior evidence given b y the prior o dds, exp erimental evidence giv en by the Ba yes factor and the predictive risk ratio which ac- coun ts for the loss of c ho osing the wrong mo del. Ac- cording to T rottini and Spezzaferri [7], the exp ecta- tions in the abov e ratio hav e the following meaning: E θ 1  KL  π ( q | θ 1 , D , M 1 ) || π ( q | D , M 2 )  - the risk of c ho os- ing m odel class M 2 when the true mo del b elongs to M 1 ; E θ 1  KL  π ( q | θ 1 , D , M 1 ) || π ( q | D , M 1 )  - even if we rep ort the distribution π ( q | D , M 1 ) when the true model b elongs to M 1 , there is a risk incurred due to the unknown v alue of θ 1 that generated the true model. Comparing with the previous model selection scheme, the follo wing informa- tion is used in this scheme to select the best mo del: QoI, mo del complexity , data fit, and prior knowledge. 4.2 Calculating the exp ected utility used in mo del selection The calculation of the QoI-a w are evidence in Eq.(10) is c hallenging as we are dealing with high dimensional in- tegrals, and the n um b er of samples in the posterior dis- tributions is dep endent on the MCMC algorithms and computational complexity of the forward mo del. Thus, w e would like to simplify this calculation. Starting from Eq.(10) the follo wing expression for the exp ected utility can b e obtained: E θ , s [ U q ( θ , s , M j )] = − K X i =1 π ( M i | D , M ) Z π ( θ i | D , M i ) π ( q | θ i , D , M i ) log π ( q | θ i , D , M i ) π ( q | D, M j ) d θ i d q = − K X i =1 π ( M i | D , M ) Z π ( q , θ i | D , M i ) log π ( q | θ i , D , M i ) d θ i d q + Z π ( q | D, M ) log π ( q | D, M j ) d q (14) Where the predictive p df under all models is giv en b y , π ( q | D , M ) = K X i =1 π ( M i | D , M ) π ( q | D , M i ) . (15) Since the first term in Eq.(14) is the same for all models M j , for j = 1 . . . K , the optimization in Eq.(9) is equiv a- len t with maximizing the second term in Eq.(14), whic h is the negative cross-entrop y betw een the predictiv e dis- tribution conditioned on all the models and the predictive distribution conditioned on the j th mo del: M ∗ = arg max M j ∈M − H  π ( q | D , M ) , π ( q | D , M j )  (16) By writing the optimization as a minimization instead of a maximization and subtracting the en tropy of the pre- dictiv e distribution conditioned on all the mo dels, then one can rewrite the mo del selection problem as, M ∗ = arg min M j ∈M H  π ( q | D, M ) , π ( q | D, M j )  − H  π ( q | D, M )  = arg min M j ∈M KL  π ( q | D, M )         π ( q | D, M j )  (17) Th us, the b est mo del to predict an unobserved quan tity of interest q is the one whose predictive distribution b est appro ximates the predictive distribution conditioned on all the mo dels. This is rather intuitiv e as all we can say ab out the unobserv ed quantit y of in terest is enco ded in the predictive distribution conditioned on all the mo dels. The predictive p df under all mo dels b eing the most robust estimate of the QoI for this problem. This model selection sc heme rev eals that when the p os- terior model plausibility is not able to discriminate b e- t ween the mo dels, the prediction obtained with the se- lected model is the most robust prediction we can obtain with one model. Thus w e are able to accoun t for mo del uncertain ty when predicting the QoI. On the other hand, in the limit, for discriminatory observ ations, when the p osterior plausibilit y is one for one of the models, the t wo selection schemes b ecome equiv alent. 5 Mo del Problem The mo del problem consists of a spring-mass-damper sys- tem that is driv en by an external force. The spring-mass- damp er and the forcing function are considered to b e sep- arate physics such that the full system mo del consists of a coupling of the dynamical system mo deling the spring- mass-damp er system and a function mo deling the forc- ing. In this mo del problem, synthetic data are generated according to a truth system. 5.1 Mo dels This section describ es the models that will form the sets of in terest for the single ph ysics. The models of the spring-mass-damp er system tak e the following form: m ¨ x + c ˙ x + ˜ k ( x ) x = 0 . (18) The mass is assumed to be perfectly kno wn, m = 1, and the damping co efficient c is a calibration parame- ter. Mo del form uncertaint y is introduced through the spring mo dels ˜ k ( x ). Three mo dels are considered: a lin- ear spring ( OLS ), a cubic spring ( OCS ), and a quintic spring ( OQS ), given by the following relations: ˜ k OLS ( x ) = k 1 , 0 (19) ˜ k OC S ( x ) = k 3 , 0 + k 3 , 2 x 2 (20) ˜ k OQS ( x ) = k 5 , 0 + k 5 , 2 x 2 + k 5 , 4 x 4 (21) The mo dels of the forcing function are denoted ˜ f ( t ). Three mo dels are considered: simple exp onen tial deca y ( SED ), oscillatory linear deca y ( OLD ), and oscillatory exp onen tial decay ( OED ): ˜ f S E D ( t ) = F 0 exp( − t/τ ) (22) ˜ f OLD ( t ) =  F 0 (1 − t/τ ) [ α sin( ω t ) + 1] , 0 ≤ t ≤ τ 0 , t > τ (23) ˜ f OE D ( t ) = F 0 exp( − t/τ ) [ α sin( ω t ) + 1] (24) The coupling of the spring-mass-damp er and the forcing is trivial. Thus, only a single coupling mo del is consid- ered, and the coupled mo del is given by m ¨ x + c ˙ x + ˜ k ( x ) x = ˜ f ( t ) . (25) There are three choices for ˜ k ( x ) and three choices for ˜ f ( t ), leading to nine total coupled mo dels. 5.2 The T ruth System T o ev aluate the tw o selection schemes: Bay esian mo del selection, and predictiv e mo del selection, we can con- struct the true system which will be used to generate data and giv e the true v alue of the QoI. The comparison of the t wo selection criteria will b e done with resp ect to differ- en t subsets of mo dels, and the abilit y of the b est model to predict the true v alue of the QoI. The true model is describ ed by OQS-OED : m ¨ x + c ˙ x + ˜ k OQS ( x ) x = ˜ f OE D (26) where m = 1, c = 0 . 1. The parameters for the spring mo del are set to k 5 , 0 = 4, k 5 , 2 = − 5, and k 5 , 4 = 1. The true forcing function is given b y the follo wing v alues for the parameters: F 0 = 1, τ = 2 π , α = 0 . 2, ω = 2. The QoI of the coupled mo del is assumed to b e the max- im um v elo cit y ˙ x max = max t ∈ R + | ˙ x ( t ) | . The observ able for physics A is given b y the kinetic energy v ersus time: 1 2 ˙ x ( t i ) 2 for i = 1 , . . . , N . Note that this contains the same information as the velocity except that it is ambiguous with respect to the sign. The observ able for physics B is giv en by the force versus time: f ( t i ) for i = 1 , . . . , M . In b oth cases simulated observ ations hav e b een gener- ated by perturbing the deterministic predictions of the true mo del, with a log-normal m ultiplicative noise with standard deviation of 0 . 1. 6 Numerical Results The inv erse problem of calibrating the mo del parame- ters from the measurement data is solved using MCMC sim ulations. In our simulations, samples from the p oste- rior distribution are obtained using the statistical library QUESO [5, 2] equipp ed with the Hybrid Gibbs T ransi- tional Marko v Chain Mon te Carlo metho d prop osed in Ref. [1]. One adv antage of this MCMC algorithm is that it provides an accurate estimate of the log-evidence using the adaptiv e thermodynamic in tegration. Estima- tors based on k -nearest neighbor are used to compute the Kullbac k-Leibler div ergence in Eq.(17), see App endix. The use of these estimators is adv antageous esp ecially when only samples are av ailable to describ e the underly- ing distributions. Three differen t scenarios are constructed to assess the predictiv e capability of the mo dels selected using the t wo selections schemes: Bay esian mo del selection and pre- dictiv e mo del selection. All the uncertain parameters of the models are considered uniformly distributed and the mo del error has also b een calibrated and propagated to the QoI. First, all the mo dels are included in the tw o sets, in- cluding the comp onen ts used to generate the true mo del. F or oscillators the mo del class set is given b y M A = { M OLS , M OC S , M OQS } and for forcing function M B = { M S E D , M OLD , M OE D } . A num b er of 10 measuremen ts ha ve b een considered for the oscillators and 61 for the forcing. T able 1 summarizes the results obtained after applying the t w o approaches. On the first column and the first row, under eac h mo del one can find the mo del plausibilit y after calibration, and the first num ber in a cell gives the plausibilit y of the corresp onding coupled mo del. The n umber in the paren thesis is the KL div er- gence used in the predictiv e mo del selection. In this case, the observ ations provided are enough to discriminate the mo dels at single physics level, oscillator OQS and the forcing OED are selected in this case. Here, the pre- dictiv e mo del selection is consistent with the plausibilit y based mo del selection. Notice that the true mo del b e- longs to the mo del class OQS-OED , and the prediction of the selected model cov ers the true v alue of the QoI, see Fig.2a. One computational adv an tage is that when dis- criminatory observ ations are av ailable, one do es not need to carry the analysis on all the coupled mo dels, just on the coupling of the b est single physics ones and still be in agreemen t with predictiv e requirements. W e argue that for very complex and hierarc hical systems, with multiple lev els of coupling, such a situation should b e preferred and exploited b y designing experiments to collect mea- suremen ts in tended to discriminate mo dels. F or the second scenario we remov e the forcing that gen- erated the true mo del. No w the sets of mo del classes are giv en b y M A = { M OLS , M OC S , M OQS } and M B = { M S E D , M OLD } . The same n umber of observ ations are T able 1: Results Case 1 SED OLD OED* 0 . 00 0 . 00 1 . 00 OLS 0 . 00 0 . 00 0 . 00 0 . 00 (3 . 75) (3 . 36) (3 . 67) OCS 0 . 00 0 . 00 0 . 00 0 . 00 (4 . 19) (3 . 91) (4 . 24) OQS* 0 . 00 0 . 00 1 . 00 1 . 00 (1 . 15) (5 . 09) (0 . 00) considered for the oscillators and 7 measuremen ts for the forcing mo dels. The results are presented in T able 2. As b efore, we are able to discriminate the oscillators, ho w- ev er we cannot say the same for the forcing mo dels. This can b e seen in Fig. 2c, where we can see the predic- tion of the observ able with the t wo forcing models after calibration. The data supp orts almost equally well both forcing functions. In this case the tw o selection schemes yield t w o different mo dels: OQS-OLD for the Bay esian mo del selection and OQS-SED for the predictiv e mo del selection, see T able 2. Lo oking at their predictions for the QoI, Fig. 2b, we see that the pdf provided b y the mo del selected using conv entional Ba y esian mo del selec- tion doesn’t ev en co v er the true v alue of the QoI, whereas the one c hosen by the predictive selection scheme cov ers the true v alue of the QoI in the tail. Thus, while the plau- sibilit y based selection ignores the model uncertain ty , the predictiv e selection approach yields the mo del with the most robust predictive p df for the QoI. This prediction incorp orates as m uch as p ossible model uncertaint y that one can obtain with just one model. Therefore, the pre- dictiv e approac h is recommended in the case when dis- criminatory observ ations are not a v ailable and one mo del has to b e chosen instead of mo del av eraging, esp ecially for complex hierarchical systems. T able 2: Results Case 2 SED OLD 0 . 44 0 . 56 OLS 0 . 00 0 . 00 0 . 00 (6 . 77) (7 . 35) OCS 0 . 00 0 . 00 0 . 00 (5 . 66) (7 . 23) OQS* 0 . 44 0 . 56 1 . 00 (0 . 62) (1 . 64) T able 3: Results Case 3 SED OLD 0 . 48 0 . 52 OLS 0 . 05 0 . 06 0 . 12 (0 . 39) (0 . 82) OCS 0 . 42 0 . 45 0 . 88 (0 . 37) (0 . 34) Lastly , we are not including in the model sets any of the components that generated the true mo del. Th us, M A = { M OLS , M OC S } and M B = { M S E D , M OLD } . In this case only 5 observ ations are considered for the oscil- lators and 4 for the forcing functions. W e can see from T able 3 that at the single ph ysics level w e are not able to discriminate the oscillators or the forcing functions. F or the coupled mo dels b oth approaches c ho ose the same mo del OCS-OLD , ho wev er lo oking at the predictions of all coupled mo dels, including their av erage prediction we see that all of them giv e very lo w likelihoo d for the true v alue of the QoI. This case emphasizes that the predic- tion approac h has the same drawbac k as the Ba yesian mo del selection and Ba yesian model av eraging. Mainly , w e are at the mercy of our h yp otheses and the only wa y to escap e this case is to generate additional h yp otheses. 0 2 4 6 8 10 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 max(v(t)) p( max(v(t)) ) Maximum velocity Osc.OQS−Force.OED Truth (a) Case 1: QoI pdf 0 1 2 3 4 5 0 2 4 6 8 10 12 14 16 18 max(v(t)) p( max(v(t)) ) Maximum velocity Osc.OQS−Force.SED Osc.OQS−Force.OLD Truth (b) Case 2: QoI pdfs 0 5 10 15 20 25 30 35 0 0.5 1 1.5 2 2.5 Calibration results time F(t) SED meas.unc. OLD meas.unc. SED 99% CI OLD 99% CI Mean SED Mean OLD Observations (c) Case 2: Observable 0 1 2 3 4 5 0 0.5 1 1.5 2 2.5 max(v(t)) p( max(v(t)) ) Maximum velocity Osc.OLS−Force.SED Osc.OLS−Force.OLD Osc.OCS−Force.SED Osc.OCS−Force.OLD Averaged Truth QoI Selection Evidence Selection (d) Case 3: QoI pdfs Figure 2: Results for the three cases considered 7 Conclusions In this paper, we hav e presented a mo del selection criterion that accounts for the predictive capabilit y of coupled mo dels. This is esp ecially useful when com- plex/m ultiphysics mo dels are used to calculate a quan tity of interest. It has been sho wn that the prediction obtain with the mo del c hosen b y the predictive approach, is the most robust prediction that one can obtain with just one mo del. This is b ecause in part it incorp orates mo del un- certain ty , while conv en tional Ba yesian model selection ig- nores it. F or discriminatory measurements the Bay esian mo del selection and predictive mo del selection are equiv- alen t. This suggests that when additional data collection is p ossible then designing exp eriments for model discrimi- nation is computationally preferred for complex mo dels. App endix The approximation of the Kullback-Leibler div ergence is based on a k -nearest neighbor approach [8]. KL  p ( x | D n )         p ( x | D n − 1 )  ≈ d X N n N n X i =1 log ν n − 1 ( i ) ρ n ( i ) + log N n − 1 N n − 1 where d X is the dimensionalit y of the random v ari- able X , N n and N n − 1 giv e the num b er of samples { X i n | i = 1 , . . . , N n } ∼ p ( x | D n ) and { X j n − 1 | j = 1 , . . . , N n − 1 } ∼ p ( x | D n − 1 ) resp ectiv ely , and the t w o distances ν n − 1 ( i ) and ρ n ( i ) are defined as follows: ρ n ( i ) = min j =1 ...N n ,j 6 = i k X i n − X j n k ∞ and ν n − 1 ( i ) = min j =1 ...N n − 1 k X i n − X j n − 1 k ∞ . Ac kno wledgments This material is based up on w ork supp orted by the De- partmen t of Energy [National Nuclear Security Adminis- tration] under Award Number [DE-F C52-08NA28615]. References [1] S. H. Cheung and J. L. Bec k. New Bayesian up- dating metho dolo gy for mo del validation and r obust pr e dictions b ase d on data fr om hier ar chic al subsystem tests . EERL Report No.2008-04, California Institute of T ec hnology ., 2008. [2] S. H. Cheung and J. L. Beck. New ba y esian up dating metho dology for model v alidation and robust predic- tions of a target system based on hierarchical subsys- tem tests. Computer Metho ds in Applie d Me chanics and Engine ering , 2009. Accepted for publication. [3] S. Geisser and W.F. Eddy . A predictiv e approach to mo del selection. Journal of the Americ an Statistic al Asso ciation , 74(365):153–160, 1979. [4] A. San Martini and F. Sp ezzaferri. A predictive mo del selection criterion. Journal of the R oyal Statistic al So- ciety. Series B (Metho dolo gic al) , 46(2):296–303, 1984. [5] Ernesto E. Prudencio and Karl W. Sc hulz. The Paral- lel C++ Statistical Library ‘QUESO’: Quantification of Uncertain ty for Estimation, Sim ulation and Opti- mization. Submitte d to IEEE IPDPS , 2011. [6] Gabriel T erejanu, Ro chan R. Upadhy ay , and Kenji Miki. Bay esian Experimental Design for the Active Nitridation of Graphite b y Atomic Nitrogen. Ex- p erimental Thermal and Fluid Scienc e, under r eview , 2011. [7] M. T rottini and F. Sp ezzaferri. A generalized predic- tiv e criterion for mo del selection. Canadian Journal of Statistics , 30:79–96, 2002. [8] Qing W ang, S.R. Kulk arni, and S. V erdu. A nearest- neigh b or approach to estimating divergence b etw een con tinuous random v ectors. In Information The ory, 2006 IEEE International Symp osium on , pages 242 –246, 2006.