Comment: Bibliometrics in the Context of the UK Research Assessment Exercise

Statistic al Scienc e 2009, V ol. 24, No. 1, 15–1 6 DOI: 10.1214 /09-STS285A Main article DO I: 10.1214/09-STS285 c  Institute of Mathematical Statisti cs , 2009 Comment: Bibliometrics in the Con text of the UK Resea rch Assessment Exerc ise Berna rd W. Silverman Abstr act. Researc h fundin g and r eputation in the UK h a v e, for o v er t w o decades, b een increasingly dep endent on a regular p eer-review of all UK departments. This is to mo v e to a sy s tem more based on bib- liometrics. Ass essm en t exercises of this kind in fluence the b eha vior of institutions, departments and individ uals, and therefore bibliometrics will ha v e effects b eyo nd simple measuremen t. Key wor ds and phr ases: Bibliometrics, researc h fun d ing, p erve rse in- cen tiv es. In the United K ingdom’s Research Assessment Ex- ercise (RAE), ev ery u niv ersit y may submit its re- searc h in ev ery discipline for assessment. On this assessmen t rests a considerable amount of fund ing; indeed a n umb er of univ ersities, leading “researc h unive rsities” in American nomenclature, gain more from this source of researc h f undin g than from go v- ernment fundin g for teac hing. Within broad su b ject bands, the Higher Ed ucation F unding Council for England fun ds teac hing o n a flat r ate p er student. So the amoun t of fu nding a student of Mathemat- ics attracts is the same w hic hev er univ ersit y they attend. O n the other hand, fu n ding for researc h is selectiv e: those departments wh ic h fare well on the Researc h Assessmen t Ex er cise r eceiv e more fun ding as a result. This is in ad d ition to an y income from gran ts and gran t o verheads. The RAE and its pred ecessors hav e b een r unning for o ve r t wo decades, a nd ha ve alwa ys b een based on p eer review, though numerical d ata on student n umb er s and grant income also hav e s ome inpu t in to the assessmen t. Ho w ev er, it is prop osed that “met- rics,” whic h includ e so-called bibliometric data, will B. W. Silverman is Master, St Peter’s Col le ge, Oxfor d OX1 2DL, Unite d Kingdom e-mail: b ernar d.silverman@sp c.ox. ac.uk . This is an electr onic reprint of the or iginal article published by the Institute of Mathematica l Statistics in Statistic al Scienc e , 20 09, V ol. 2 4, No. 1, 15– 16 . This reprint differ s from the original in pag ination and t yp ogr aphic detail. b e the main part of the system whic h w ill so on s uc- ceed the RAE, though it is probable that in mathe- matical sub jects, p eer review will con tinue to p la y a considerable part. Th e details hav e yet to b e work ed out. In the 2008 RAE, I was c hair of the committee whic h r eview ed Probabilit y , Statistic s and the more mathematical asp ects of Op erational Resea rch. The committee’s exp erience of conducting the assessment as a whole strengthened our view that p eer review m ust b e at th e core of an y fu ture assessmen t of researc h in our area . Reliance on bibliometric and purely qu antitat iv e metho ds of assessment w ould, in our u nanimous view, introd uce serious biases, b oth in to the assessment pro cess and , p erhaps more se- riously , into the b ehavio r of institutions and of in- dividual researc hers, to the d etrimen t of the v ery researc h which the exercise is intended to supp ort. It is imp ortant to stress the effect of an y system on the b ehavio r of institutions. Th e curren t p eer-review RAE has had clear effects on ins titutional b eha vior, some of them certainly p ositiv e, some of them p er- haps less so. F or example, the RAE gives explicit adv an tages to new entran ts to the pr ofession; those en tering in the last few years are allo w ed to subm it a sm aller corpu s of work for assessment, and ther e is also credit giv en w ith in the p eer review s ystem for a sub jectiv e assessmen t of the general vitalit y of the department. Of the appr o ximately 400 researc h - activ e facult y declared to the statistics panel in the 2008 RAE, ab out a qu arter w ere new entran ts since 2001, and the RAE h as certainly give n an imp etus 1 2 B. W. SIL VERMAN to this new recruitment, as it also do es to the mo- bilit y of leading researc hers b et w een institutions. On a more negativ e note, the fixed date of the assess- men t encourages a “b o om-bus t” men talit y , w here some institutions hire in considerable n umbers of new facult y in the p erio d leading up to the censu s date; to make up for this extra exp end iture, during the p erio d after th e census date there is something of a moratorium on app oin tments. Th e consideration of grant income in the RAE giv es extra gearing to the p ressure on facult y to pursu e grant-supp orted researc h rather than to wo rk in a more ind ividual fashion. There can b e little doubt that a stronger empha- sis on bibliometrics (and other “metrics”) in assess- men t exercises will affect institutional b ehavio r, es- p ecially in systems where assessment results ha ve b oth reputational and fi scal impact. Because indi- viduals are sensitive to institutional pressures, they to o will mo dif y their b eha vior in resp onse. F or ex- ample, it is probab ly the case that there is a high correlation b et w een h -ind ex (sa y) and p erceiv ed qual- it y and reputation of researc h er s . Similarly , highly- cited pap ers are almost alw a ys infl uent ial and im- p ortant (though th e conv erse is not necessarily true). Ho wev er, basing jud gmen t of individu als or depart- men ts on citatio n coun t rather than s ome assess- men t of u nderlying quality w ou ld ha ve the obvio us p erverse consequences. P erh aps the obvious anal- ogy wo uld b e with a system that coun ts pub lica- tions: of course there is some correlatio n b et ween the o verall qualit y of a researc her’s work and th e n um- b er of pap ers she or he pu blishes, b ut the “publish or p er ish ” ment alit y engendered by simple p ap er- coun ting militates against the careful and thought - ful researc her who only writes pap ers w hen they feel they ha v e something very serious to sa y , or—wo rse still—writes b o oks rather than pap ers. P erhaps the bibliometric ve rsion is “b e cite d or b enighted”? One of the argumen ts the UK universit y fund - ing agencies u s ed initially in fav or of b ibliometrics w as that, when aggregat ed o ve r wh ole un iv ersities, the resu lts of “metrics-based” assessmen ts were very highly correlated with p eer-review assessmen ts. As statisticia ns, we should b e wel l placed to refute the fallacy of th is argumen t. It mak es complete sense that a strong u niv ersit y will ha v e more than its fair share of highly-cited researc h ers r igh t across the range of disciplin es. An y errors and biases w ill to some exten t a verage out. But d isaggregation do wn to de- partmen ts, and ev en individu als, encourages the elim- ination, or do wngrading, of disciplin es and sub-disciplines w h ic h d o not generate large amoun ts of citations. Within d isciplines, there is a risk of un- derv aluing individuals wh ose work is deeply infl uen- tial in wa ys th at do not sho w up in short-term cita- tion counts. And man y individual r esearchers would no dou b t b o w to p erceive d pressu re to b e “cited or b enighte d.” If citation coun ts are unreasonable, the u se of im- pact factors seems almost in defensible. Assigning a notion of qualit y to a p ap er on the grounds of th e impact f actor of a journal is like assigning a notion of wea lth to an individual on the basis of the a v erage GDP of their home coun try . Man y c h ildren gro wing up in England in the 1950s w ere under the impres- sion that all Americans we re w ealth y! Of course, if one kn o w s ab out the refereeing standards of a lead- ing journal, it m a y , or ma y not, b e r easonable to supp ose that if a p ap er has p assed these s tand ards it has a go o d c hance of b eing of high qualit y , but that is a v ery differen t thing f rom assessing the journal b y an imp act factor. In conclusion, I wo uld v ery strongly sup p ort the underlying thesis of the pap er: citation statistics, impact factors, th e wh ole p araphernalia of biblio- metrics ma y in some circumstances b e a useful ser- v an t to us in our researc h . Bu t they are a v ery p o or master ind eed.