Rejoinder: Citation Statistics

Statistic al Scienc e 2009, V ol. 24, No. 1, 27–2 8 DOI: 10.1214 /09-STS285REJ Main article DO I: 10.1214/09-STS285 c  Institute of Mathematical Statisti cs , 2009 Rejoinder: Citation Statistics Rob ert Adler , John Ewing and P eter T a ylo r W e would lik e to thank the discussan ts for reading our rep ort and for their insigh tful and constru ctiv e commen ts. T o start our brief resp onse, we would lik e to qu ote Bernard Silv erm an’s p hrase “redu cing an assessmen t of a n individual to a single num b er is b oth morally and professionally repugnant.” Bernard puts it strongly , but his underlying p oint, with wh ic h we strongly agree, is that “researc h qualit y” is not some- thing that ought to b e regarded as w ell-ordered. W e note the general supp ort for the case that any analysis should b e carried out in the con text of a prop erly-d efined mo del. P eter Hall calls for statisti- cians to un dertak e a study of “the n ature of citation data, the in formation they contai n and metho ds for analysing them if one must.” Among the th ree of us, there are v arying leve ls o f enth usiasm for advo- cating suc h a pro ject. A p ossible do wnside is the danger th at such a s tudy will add to the bu r geoning n umb er of prop osals f or carrying out citation anal- ysis in a “b etter” w a y , and none of us ha ve muc h en th usiasm f or this. O n the plus side, su c h a study w ould enable the mathematical sciences comm un it y to comment more authoritativ ely on citation statis- tics and the quantita tiv e ranking measures th at are deriv ed from them. Giv en that the scien tometric in - dustry sho ws every sign of gro w ing, it can b e argued that it is the resp ons ibilit y of th e mathematical sci- ences, and p articularly of statisticians, to develo p this capabilit y . R ob ert A d ler, F acult y of Ele ctric al Engine ering, F aculty of Industrial Engine ering and Management, T e chnion, Haifa, Isr ael, 32000 e-mail: r ob ert@ie ad ler.te chnion.ac.i l . John Ewing, Pr esident, Math for Ameri c a 800 Thir d Ave, 31st fl , New Y ork, New Y ork 10022, U SA e-m ail: ewing@mathfor americ a.or g . Peter T aylor, Dep artment of Mathematics and Statistics, University of Melb ourne, Vic 3010, Austr alia e-mail: p.taylor@ms.unimelb.e du.au . This is an ele ctronic repr int of the original article published by the Institute o f Ma thema tical Statistics in Statistic al Scienc e , 200 9, V ol. 2 4, No. 1, 27 –28 . This reprint differs fr om the original in pag ina tion and t yp ogr aphic detail. Da vid Spiegelhalter and Harv ey Goldstein p oin ted out that there is a lac k of indep endence b et ween in- dividual authors’ citation r ecords due to issues of co- authorship. The effects of this lac k of indep end en ce seem to b e v ery p o orly un dersto o d, an d nothing in the literature that we review ed sheds any ligh t on them. In our rep ort, we sp en t some time discussin g the meaning of citations. Sune Lehmann , Benny Lautrup and And rew Jac kson to ok this p oin t further in their discussion of the fact th at there needs to b e agree- men t on the basic meaning of a researc her’s citation distribution, whic h is something that go es b eyond merely kno wing wh at citations mean, whic h itself is not clear. Their example inv olving researc hers A and B mak es this p oint clearly . W e wo uld like to emphasise thr ee fi nal p oin ts that ha v e more to do with h uman b eha vior than statis- tics, and w hic h were not emph asised in the rep ort itself. The first is related to Bernard Silve rman’s p oint that an y measuremen t or ranking system will driv e researc her b ehavio r via natural feedbac k mec h- anisms. T r aditionally , the mechanisms adopted in academia ha v e b een qu alitativ e rather than quan- titativ e. Peer review has b een at the core of the system. When carefully done, p eer r eview not only pro vides accurate and p rofessional assessments of an individual’s contributions, bu t it also pr o vides a bal- anced and educated inte rpr etation of quanti tativ e information su c h as p rizes and citation data. Mo v- ing to a system based purely on quanti tativ e citat ion metrics will deliver feedb ac k more frequ en tly , more unequivocally , and in a d ifferen t w a y . It is not at all clear that “go o d researc h” (and we realise how loaded this term is) will b e encouraged b y such a system. Our strong opinion is that this feedbac k as- p ect is very imp ortan t. Related to this issue is another of particular con- cern. In general, it is n ot all that easy to f o ol one’s p eers, bu t it take s little imaginatio n to see how, b y adopting citation p olicies that are different fr om the norm in a particular d iscipline or sub-discipline, a small group of individuals could easily fo ol an au- tomated assessment system built on citation data. Assessmen t is imp ortan t to all of us , as individuals, 1 2 R. ADLER, J. EWIN G AND P . T A YLOR as institutions, and as r epresent ativ es of disciplines. Adopting a system, for short term gains, th at is so easily op en to abuse is a r isk to researc h standards in the long term. Our final p oin t, wh ic h h as b een amplified by our exp eriences since the rep ort wa s first released, is that almost ev ery one is affected by conflicts of in ter- est when the topic of researc h assessment comes u p. F or most of us, the wa y our r esearc h is regarded go es to the ve ry core of our professional iden tit y , and it w ould b e a rare in dividual who could isolate his or her opinion ab out a particular metho d of research assessmen t and the wa y that his or her o wn research is r ank ed b y the metho d. F or example, most p eople who d o well according to h-indices tend to think that the h -index is not a bad measure of r esearc h er qualit y . There are also individ uals who hav e bu ilt ca- reers, and companies that ha ve profited, from under- taking researc h assessment in a particular wa y . S ince w e are certain that it is health y for all d isciplines to ha v e a multitude of skills and temp eraments in their researc h communities, this observ ation leads u s bac k to where we started: “researc h qualit y” is an inher- en tly m ultidimensional ob ject and should b e treated as suc h.