Bootstrapping Confidence Levels for Hypotheses about Quadratic (U-Shaped) Regression Models

B O OT S T RA P P IN G C O N FI D E N C E LE V E L S FO R H Y PO T H E S ES A B O U T Q U AD R A T IC ( U - S H AP E D ) R E GR E S S IO N M O D EL S 12 June 2012 Michael Wood University of Portsmouth Bu siness School SBS Department, Richmond Building Portland Street, Portsmouth PO1 3DE, UK +44(0)23 9284 4168 michael.wood@port.ac.uk . Abstract Bootstrapping can produce confidence levels for hypotheses about quadratic regression models – such as whether the U-shape is inverted, and the location of optima. The meth od has several advantages over conventional methods : it provides more, and clearer, inf ormation, and is flexible – it could easily be app lied to a wide variety of different types of models. The utility of the method can be enhanced by for mulating models with interpretable coefficie nts, such as the location and value of the o ptimum. Keywords: Bootstrap resam pling; Confidence level; Quad ratic model; Regression, U-shape. Bootstrappin g confidence l evels for hy potheses abo ut regr ession models 2 B O OT S T RA P P IN G C O N FI D E N C E LE V E L S FO R H Y PO T H E S ES A B O U T Q U AD R A T IC ( U - S H AP E D ) R E GR E S S IO N M O D EL S Abstract I show how bootstrapping can produce confiden ce levels for hypotheses about quad ratic regression models – such as wheth er the U-shape is inverted, and the lo cation of optima. The method has several advantages over c onventional methods: it provides more, an d clearer, information, and is flexible – it could easily be applied to a wide variety of differen t types of models. The utility of the method can b e enhanced by formulating m odels with interpretable coefficients, such as the location and value of the optimum. 1. Introduction Quadratic (U-shaped) models are widely used in management and economics – e.g. in research on well-being (Blanchflower and Oswald, 2 008), staff turnover (Glebbeek and Bax, 2004) and th e environmental Kuznets curve (Dinda, 2004). They are conventionally anal yzed by reporting regression coefficients, and p values, f or the independent variable and its square. However, there is a very large literature on the proble ms with p values in general terms (e.g. Nickerson, 2000), and they are particularly problematic f or hypotheses about U-shapes: ther e are two p values which makes it difficult to get a clear idea of the degree of support for the U-shape hypothesis. This paper shows how simp le bootstrapping can be used to derive confidence le vels for hypotheses about quadratic regression m odels. The approach is intuitive, flexible, avoids the distributional assumptions required by con ventional methods, and provides users with dir ect estimates of the (posterior) probabilities of the competing hypotheses. I also suggest reformulating the standard model in ter ms of more easily interpreted co efficients. Most applications of bootstrapp ing use relatively complex, possibly novel, approaches to intractable problems. This paper differs in tw o respects: first, the bootstrap method used is a Bootstrappin g confidence l evels for hy potheses abo ut regr ession models 3 very simple one, and second , the problem tackled is widely seen as solved. The originality of this paper lies in showing how a very simple method can provide better ans wers to the problem than the standard ones. Figure 1 shows one model d iscussed by Glebbeek and Bax (2004). (Glebb eek and Bax do not show this diagram. Dr. Glebbeek, howe ver, was kind enough to give me acce ss to the data.) They investigated the hypothesis tha t there is an “inverted U-shape relationship” between two variables – staff turnover and organizational performance. The hypothesis is base d on the idea that both high and low levels of staff turn over are likely to lead to poor perfor mance, so there will be an optimum level of turno ver (e.g. 10% of the staff leaving in a year) with performance falling off above and below this. For an inver ted U shape the regression coefficie nt for the squared turnover term shou ld be negative, and for the linear term the coefficient should be positive. In the model shown in Figure 1 the sig ns of the coefficients were consistent with this hypothesis but neither was signi ficant ( p >0.1 in both cases). Figure 1. Predicted performance from qua dratic model (after adjusting for val ues of three control variables) (The solid lin e is the pred iction from t he regression model; the scattered points are t he data on which th e regression mo del is based.) -50000 0 50000 100000 150000 200000 250000 0 10 20 30 40 Performance Staff turnover (% per year) Bootstrappin g confidence l evels for hy potheses abo ut regr ession models 4 2. Rewriting the model with easily int erpreted coefficients The equation of the curve is usually written as: y = ax 2 + bx + c where x is the independent variable (turnover), y is the dependent variabl e (performance), and c is constant (and includes the adjust ment for the control variables). Alternativel y, exactly the same curve can be written as y = M + U ( x – L ) 2 The square term in this equation must be zero or positive, so M is the maximum value of y if U <0 and the minimum if U >0. L is obvi ously the location ( x coordinate) of this maximum or minimum, and U is the degree of curvature with positive values corresponding to an upright U- shape and higher values (positively or negati vely) indicating a greater degree of curvature. Comparing the two equations: U = a, L = - b /2 a , M = c – b 2 /4 a . In Figure 1, M = 69575, L = 6.3% and U = -8 6.7. Each coefficient then has an obvious interpretation. Obviously, for an inverted U-shape, L >0 (since the turning point must occur for positive x ) and U <0. 3. Bootstrapping confidence levels fo r hypotheses The question now arises of whether this de monstrates that a similar pattern woul d occur if the analysis was done with the whole population from which the sample was drawn. Conventionally this question is answered by testing two nul l hypotheses – the first being that the coefficient of the linear term is zero, and the second being that the squared term is zero. In the model represented by Figure 1, neither coefficient is signific antly different from zer o. The evidence provides some support for the inverted U-shape hypothesis, but it is difficult t o combine the two p values into a single figure to indicate the overall strength of the support for this hypothesis. A bootstrap method provides both a way out of this difficulty, and an easy approach to other questions we are likely to have. The result from the bootstrap analysis bel ow is that the data on which Figure 1 is based suggests a c onfidence level, or probability, of 67% for the Bootstrappin g confidence l evels for hy potheses abo ut regr ession models 5 inverted U shape hypothesis. The method is implemented on an Excel spreadsheet (available on the web) which can easily be adapted to anal yze different datasets or models. Bootstrapping involves taking a large number of resamples (say 1000) with replacement from a sample of data. (For each r esample we choose a member of the original s ample at random, then replace it and choose again, until we have a sample the sa me size as the real sample. This means that some memb ers of the original sample will appear in the resample more than once, and others not at all.) Th e simplest approach – which I use here – is t hen simply to assume that each resample represents one possible distribution for the origin al population, and that the collection of 1000 resamples represent s a probability distributi on for these possibilities. I will consider the justification for this a ssumption after I have explained how it w orks in our example. Figure 2. Predicted performance using a quad ratic model (after adjusting for va lues of three control variables) – from the da ta (bold) and three resamples (dotted l ines) Figure 2 shows results from three resamples as w ell as the original sample. Eac h of the dotted lines in this figure is based on identical formula e to the solid line representin g the real sample, but using the data from a resa mple, rather than the original sample. Tw o of these resamples are obviously an inverted U shap e; the third is not. -20 0 0 00 -15 0 0 00 -10 0 0 00 -5 00 0 0 0 5 00 0 0 10 0 0 00 15 0 0 00 0 10 2 0 30 4 0 50 Sta ff t urnove r (% pe r ye a r) Perfor m ance Bootstrappin g confidence l evels for hy potheses abo ut regr ession models 6 These results come from an Excel spreadsheet at http://woodm.myweb.port.ac.u k/BRQ.xls . The Resample sheet of the spreadsheet all ows users to press the Recalculate button (F9) and generat e a new resample and line on the graph. It is then a simple matter to produc e more of these resamples and c ount up how many of them are inverted U-shapes. The conclusi on was that 67% of 1000 resamples g ave an inverted U shape (with L >0), which suggests tha t the confidence level for this hyp othesis, based on the data, should be put at 67%. (The same method and spreadsheet can als o be used to produce confidence bands on the grap h.) 4. Assumptions and interpretatio n The assumption made above that e ach resample represents a different po ssible population from which the original sample was drawn is just an assu mption. Resampling with replacement simulates the process of drawing samples fr om an infinite population with the sa me distribution as the original sample, so the resamples represent different samples that might have been drawn from this population . For example, the t op resample in Figure 2 – the one which is a non- inverted U-shape – shows th at even if the overall population pattern shows an inverted U-shap e this may not be true of individual sa mples. The discrepancies between the dotted lines and the bold lines in Figure 2 illustrate the extent of sa mpling error, so from this point of view it see ms reasonable to assume that we can reverse this argu ment and assume that the d otted lines represent different populations from which the sample might have been drawn. There is a large literature on testing various elaborations on the basic b ootstrap method in various different situations, with conclusi ons that are predictably mixed. Howe ver, experience shows that in “well-behaved” situations the bootstrap approach gives similar res ults for confidence intervals to stand ard methods based on probability theory. For exam ple, 1000 bootstrap resamples gives a 95% confidenc e interval for the slope of the linear model (d erived from the data in Figure 1) of –3147 to –7 04; the corresponding estimate from the formula e based on the t distribution b uilt into Excel is –3060 to –495. In this case the bo otstrap interval is 5% narrower (and the next bootstrap interval was 3% wider). Furthermore, confidence intervals deri ved by conventional, frequen tist methods are often identical to Bayesian credible inter vals based on flat prior distributions (Bayarri and Bootstrappin g confidence l evels for hy potheses abo ut regr ession models 7 Berger, 2004). This suggests that it is r easonable to interpret bootstrapped confid ence levels in probabilistic terms as Bayesian posterior probabilities, making the assumpti on that the prior distribution is uniform in the sense that each of resample results, like the four sh own in Figure 2, are equally likely. 5. Confidence levels for other hypoth eses It is very easy to use this method to obtain confidenc e levels for other hypotheses. The proportion of resamples exh ibiting any specified property can easily be worked ou t. For example, we might want to know the location ( L ) of the optimum staff turn over. The point estimate from the regression shown in Figure 1 is that the optimum perf ormance occurs with a staff turnover of 6.3%. The 1000 resamples gi ve these confidence levels for three hypotheses: • Confidence in hypothesis that L is between 0% and 10% = 30% • Confidence in hypothesis that L is between 1 0% and 20% = 37% • Confidence in hypothesis that L is above 2 0% = 0% Alternatively we might decide that an in verted U-shape needs a minimum differ ence between the optimum performance ( M ) and the value corresponding to a turnov er level of zero. Setting this minimum to 10,000 units, the confidence level then comes t o 40%. Another hypothesis of interest to Glebbeek and Bax (2004) is that the relati onship between performance and turnover is negative – this being the rival to the inverted U shape hypothesis. The top resample in Figure 2 illustrates th e importance of defining this clearly : this shows a negative relationship for lo w turnover, but a positive relation for higher turnover. If we define a negative relationship as one which is not an inverted U shape, and for which the predicted performance for Turnover = 2 5% is less than the prediction for Turn over = 0%, then the spreadsheet shows that • Confidence in hypothesis that the relati onship is negative = 33% In fact, all 1000 resamples gave either an inverted U shape or a negati ve relationship in this sense. Bootstrappin g confidence l evels for hy potheses abo ut regr ession models 8 This result gives a very different pic ture from a standard linear regression analys is. The linear regression coefficient is negative with a p value of 0.007: this is equivalent to a confidence level for a negative relationshi p of 99.6%. The reason for the difference is that ea ch method has a different underlying model (quadratic in the first case, linear in the second) and defines a negative relationship differently: the bo otstrap method has the advantage that i t can be used with any definition of a negative relationship. 6. Conclusions I have shown how bootstrap ping can be used to estimate confidence levels for whatever hypotheses are of interest. In the example, instead of two p values which are difficult to interpret, the conclusion is that we can be 67% confident about the inverted U-shape hypothesis, and 37% confident that the opti mum turnover is more than 10%. Furthermore, the bootstrap method provides an in tuitive idea of sampling variability, and the resul ts can be interpreted as probabilities. The transpar ency of the approach can also be enhan ced by reformulating models in terms of the l ocation and value of the optimum, and the degree of curvature (instead of the conventional coefficient s for the linear and square term s). This is just one example. Quadrat ic models are widely u sed, and similar methods could easily b e used to analyze hypotheses about oth er models. Acknowle dgment I am very grateful to Dr. Arie Glebbe ek for making his d ata available to me. References Bayarri, M. J., & Berger, J. O. (2004). Th e interplay of Bayesian and frequentist analysis. Statistical Science, 19 (1), 58-80. Blanchflower, D. G., & Oswald , A. J. (2008). Is well-being U-shaped over the lif e cycle? Social Science & Medicine , 66 , 1733-1749. doi :10.1016/j.socscimed.2008.01.030 Dinda, S. (2004). Environmental Kuznets curve hyp othesis: a survey. Ecological Economics , 49 , 431-455. doi:10. 1016/j.ecolecon.2004.02.011 Bootstrappin g confidence l evels for hy potheses abo ut regr ession models 9 Glebbeek, A. C., & Bax, E. H. (2004). Is high e mployee turnover really harmful? An empirical test using company records . Academy of Ma nagement Journal, 47 (2), 277-286. Nickerson, R. S. (2000). Null hypothesis significance te sting: a review of an old an d continuing controversy. Psycholo gical Methods, 5 (2), 241-301.