The Wisdom of Citing Scientists

Accepted for publication in the Journal of the American Society for Information Science and Technology The Wisdom of Citing Scientists Lutz Bornmann* & Werner Marx** *First author and corresponding author: Division for Science and I nnovation S tudies, Administrative Headquarters of the Max Planck Society, Hofgartenstr. 8, 80539 Munich, Germany. E-mail: bornmann@gv.mpg.de **Second author: Max Planck Institute for Solid State Research, Heisenbergstraße 1, D-70569 Stuttgart 2 Abstract: This Brief Communication discusses the benefits of citation a nal ysis in resea rch evaluation based on Galton's "Wisdom of Crowds" (1907). C itations are based on the asse ssment of many which is why the y can be ascribed a certai n amount of accuracy. However, we show that citations are incomplete assessments and that one cannot assume th at a high number o f citations correlate with a high level of us efulness. Only when one kn ows that a rarely cited paper has been wid ely read is it possible to say – strictl y speaking – that it was obviousl y of little use for further research. Usin g a compariso n with 'like' data, we tr y to determine that cited reference analysis allows a more me aningful analysis of bibliometric data than ti mes- cited analysis. Key words: Bibliometrics; Cited reference anal ysis; Wisdom of crowds 3 Introduction The evaluation of research is the b ackbone of m odern science: it would not be possible to determine whether a piece of research is of high or low quality without a review by a researcher's p eers (Bornmann, 2011 ). The importance of th e evaluation of research manifests itself firstly in the fact that since around the middle of the 17th century t he development of modern science in so ciet y has been closel y associ ated with the establishment of the p eer review p rocess (de Bellis, 2009). It seems th at mo dern science requires a formal proc ess with which to evaluate scientific work so that knowledg e can continue to progress. The importance of the evaluation of research is also underlined by R obert K. Merto n's inclusi on of "organised scepticism" as a key no rm in the ethos of science (Merton, 1973): "Or ganized skepticism involves a latent questioning of c ertain bases of established routine, authority, vested procedures and the realm of the 'sacred' generall y" (p. 264). Alon g with communalism (making knowledge public), universalism (rec ognition of knowl edge irrespective of it s source) and disinterestedness (not working for profit), organised sc epticism, according to Merton (1973), should be a guiding principle for researchers. While peer review has been used exclusively to assess research in almost every sci entific discipline, since the mid-1980s the use of indicators to evaluate rese arch has become increasingly important – at least in the natural sciences. Bibliometrics in particular has become a focus of int erest as an indic ator-based method of assessment. Although the strengths of each appro ach mean that bibliometri cs is primarily suit ed to the assessment of large units (such as instituti ons or countries) and peer r eview to ev aluation on a smaller scale (such as research proposals or manuscripts), info rmed peer review is seen as the ideal way forward for evaluation in research (Bornmann, 2011). With thi s approach, results based on 4 evaluations with indicators form a solid basis on which experts in the peer review pro cess can take their decisions. Bibl iometrics (or its most important tool, citation analysis) is primarily of benefit when evaluating l arge unit s because it is based on the jud gement of a lar ge numbe r of scientists. A published work contains research results which (1) can be accessed by any other member of the scientific communit y , (2) these scientists can use for thei r o wn research and (3) they can cite in their own publications. A large number of citations of a publication indicates that it has been interesting and useful for a large number of researchers (Bornmann & Marx, 2013a). In thi s Brief Communications , we point out (1) the high accuracy level of citations, because citations are based on the opinions of a large number of scientists (“Wisdom of Crowds”). Since citations are count data and not ordinal data, we argue (2) that citations are also an incomplete judgement. As count data, we think that (3) citations, in the form of cited references, say more about the citing unit than citations about the cited unit. The cited reference analysis should therefore be preferred to the times cited a nal ysis. The Wisdom of Crowds Citations measure an aspect of scientific qu ality – the impact of publi cations (van Raan, 1996). Martin and Irvine (1983) distinguish between thi s aspe ct (“the ‘ impact ’ of a publication describes its actual influence on surrounding research activitie s at a given time,” p. 70) and ‘ importance ’ (“the influen ce on the advance o f sci en tific knowledge,” p. 70) and ‘ quality ’ (“how well the research has been done,” p. 70). They consider the impact as the most important indicator of the significance of a publi cation for sci entific activities. Cole (1992) 5 sees citations as a valid indicator of quality, as the y correlate with other qualit y indicators (see also Bornmann, 2011 ; Smith & Eysenck, 2002). The benefit of citation analysis for the evaluation of research is bas ed on what Galton (1907) at the beginning of the 2 0th century called the "wisdom of crowds". Galton (1907)'s reports on the results "of a contest at an Engli sh livestock show where contestants were asked to guess the wei ght of a publicl y displayed ox . After sorting the 787 entries b y weight, Galton found that the median estimate of 1,207 pounds differed from the true w eight of 1,198 pounds by less than 1%" (Herron, 2012, p. 2278). When a large numbe r of people make a judgement or an estimate, we can ex pect it to be valid . Citation anal y sis is also based on the jud gement of many peopl e: the scienti fic community cites a paper when it has turned out to be useful, or not when it is less useful. Galton (1907)'s premise on the validit y of such crowd judgements forms the basis for Surowiecki (2004) 's book "The Wisdom of Crowds". However Surowiecki (2004) ar gues that one shoul d not assume that all crowds are wise. Crowds are onl y wis e and their judgement only achieves a high lev el of accuracy when the individual judgements are based on opini ons which were delivered independently. If w e were to apply this prov iso of Surowiecki (2004)'s to bibliometrics, we could also assume that citation analysis allows a high level of a ccuracy: firstl y, the citations are based on the opinions of a large number of scientists. Secondly, w e can assume that most citations are based on the independent opinion of the citing researcher. There are just a few case s in which publishers or reviewers have asked that citations be made, that they be discussed with colleagues or not used for immaterial considerations. However, unlike assessments of the weight of an ox, citations represent an incomplete judgement. A citation denotes the usefulness of a paper, but it is not possible to use citations 6 to ex press a graduated as sessment. Citations are c ount data (there is or there is not a citation) and not ordinal data which would conve y better or worse , important or unimport ant. Furthermore, a citin g res earcher onl y expresses w hich publi cations he or s he has found useful and does not indicate w hich publi cations were irrelevant to the writing of the paper. We cannot, therefore, automatically assum e when a publication is not ci ted that it is not important to research. W e could only assume that an uncited publication was not particularl y useful for research if we knew that it had been widely read. On the other hand, a publication that has gone largely unnoticed has few opportunities to be cited – irrespective of its quality. Nevertheless, we would not like to interpret the classification of citations as count data and the therefore accompanying disadvantages in the evaluation of public ations as meaning that citations have no use in the "wisdom of peer s" res earch evaluation. As there are two perspectives to citation analysis – citing and cited – we would li ke to posit the provocative claim in this paper th at citations make a powe rful statement p articularly in one of thes e perspectives – the cited p erspective. Citations, in the f orm of cited references, sa y more about the citing unit than citations about the cited unit. To illustrate this provocative claim, w e want to compare citations with the 'like' buttons which are used in social netwo rk services, blo gs, Internet forums and s o on to explain the preference for the cited p erspective in citation analysis. Users click 'like' buttons t o indicate that they approve of content (such as a book or a piece of music). Similarly to the times -cited data provided by Thomson Reuters in the Web of Science (WoS), Internet se rvices give the number of user s who like certa in c ontent on the basis of this information. This analogy between 'likes' and citations does not fo cus on the cognitive substance, but the measurement category of the data. W hereas the measurement category is ver y similar (b oth are count data), 7 the cognitive substance is very different. As concept symbols (Small, 1978 ) citations have a significant greater substance than 'likes' which express a certain preference onl y. The quantity of users of a product expressed b y 'likes' sa ys little about how much a content is generally approved o f. (1) As count data, 'likes' do not permit a graduated assessment. (2) Because it is not known how many us ers are familiar with the content, but have not evaluated it, knowing the numb er of users who 'like' it tell s us li ttle. It is onl y the proportion of users wh o like the content among all the users who have delivered an assessment which would allow a comparison of the popularit y of different content. Unlike t he citation data in bibliometrics, the 'like' data is used as a rule not to assess the content, but to assess the users who have made a statement about certain content. The 'like' data is therefore evalu ated for the person who has provided information and not the content which is being evaluated. The user - sp ecific statements on various content s can be used to compile a user profile which provide s information about a user's preferences. This data is of great significance for tar geted, individualised advertising. Cited reference analysis as a tool for research evaluation Nor, as a rule, is it know n for an academic public ation how man y scientists have read it, but have not considered it v ery useful (and h ave no t cited it for thi s r eason). The 'times-cited ' information is therefore only of limited significance. Reference sets are used to achieve standardisation by c omparing citation impact across diff erent fie lds (Bornmann & Marx, 2013a) but the info rmation about how many s cientists have not found the publi cation very useful is not available. Similarly to the 'like' bu tton, it would also be possi ble to argue that citations might be better used to character ise the citing unit. 8 References (citations) consist fundamentally o f three pieces of information: the author(s) of the cited paper ( author name), the jou rnal in which the cited paper appears (including information such as volume, issue and pages) and its year of publi cation. A reference analysis can refer to any one of th ese piec es of information. More data can be included: for example, it is possible to find out for each journal which publishi ng house it is iss ued b y, whether it is an open access journal or w hether it operates using the peer review process. For r esearchers , it is possible to determine w hich journals or publishing houses the y refer to in their publi cations and whether the y use primarily quality-assured papers in their work (that is, papers from journals which operate a peer review process). Further information about the affiliation of th e authors can be used for bibliometric anal y sis at instit utional or country level. The anal ysis o f the publication y ears could reveal the extent to which researchers read the current literature in their field, and whether they refer back to the historical roots of their discipline. However cited reference analysis can be used for more than compili ng a profile of the citin g scientist; it can also be used for compiling profiles of other citing units. These might be research groups, institutions or countries, or even disciplines or researched subjects. Bornmann and Marx (2013b), for example, have carried out a cited reference anal ysis on the subject of research into Aspirin®. First, they selected all the papers dealing with Aspirin® as a s pecific research topic. Then they extracted all the cited references from this fie ld - sp ecific publication set and analysed which papers, authors and journ als were cited most often. In other words: th ey categorised on the basis of the cited references rather than on the cited papers in this specific field. C ited refere nce analysis characterised research on Aspirin® b y identifying, for example, the journals preferred by the citing scientists. 9 To illustrate the difference between traditional citation analysis (ti mes-cited) and reference analysis (cited reference counts) we would like to use a d ynamically growing area in whi ch currently ver y intensive research is being und ertaken: r esearch associated with graphene, an allotrope of carbon (G eim & Kim, 2008 ; Geim & Novoselov, 2007). The papers published in 2010 with the word 'graphene' in the title were selected in the Science Ci tation I ndex (SC I). The SCI is made available for such analyses by the data provide r STN International. Table 1 shows the bibliographic data of the publications on graphene research from 2010 from a forward view (times-cited) and a backward view (cited reference counts). The most -highl y cited papers from graphene research are shown on the left -hand side and the papers most - highly cited in graphene research are shown on the right-hand side. While the right-hand side shows which publications have be en particularl y sig nificant for research associated with graphene, we onl y see from the data on the left -hand side that a number of graphene publications were significant for the research in general. However, we do not know for which research the publications were so significant. Cited reference analyses are uncommon in citation impact measurement b ut they are used for techniques such as bibliographic coupling (Ke ssler, 1963) or c iting-side journal mapping (Leydesdorff, 1994). On e of the rare examples are the cited referenc e anal yses for the “science and e ngineering indicators” (National Science Board, 2012): the share of world citations is shown for a series of sel ected countrie s and specific times. Another example is the study by Bornmann, de Mo ya-Anegón, and Le ydesdorff ( 2010) on the "Ortega Hypothesis". This study shows that high-impact research uses earlier high-impact research more than medium-impact research does. 10 The usefulness of the cited reference perspective can be just ified by the fact that many bibliometric studi es (and beyond) contain a note indicating that the citation impact of a pap er, scientist or group of scientists should be or is measured in a particular field. For ex ample, “the result is the identification of high performers within a given scientifi c field” (Froghi et al., 2012, p. 321) . “ Ideally, a measure would r eflect an individual’s relative contribution within his or her fi eld” (Kreiman & Mauns ell, 2011 ) . “That is, an account of the number of citations received by a scholar in articles published by his or her field colleagues” (Di Vaio, Waldenström, & W eisdorf, 2012, p. 92). The well-known philosopher of science an d American historian Thomas S. Kuhn formulated: “For a scientist, the solution of a difficult conceptual or inst rumental puzzle is a principal goal. His success in that endeavour is rewarded through reco gnition b y other members o f his prof essional group and b y them alone” (Kuhn, 1970, p. 21 ). Ho wever, with the standard ti mes cited anal ysis, it is not just the citation impact on a specific field that is measured, but the impact in the whole scientific community. Discussion This Brief Comm unication discusses the ben efits of citation anal ysis in research evaluation based on Galton's "Wisdom of Crowds" (1907). C itations are based on the assessment of many which is wh y th ey c an be ascribed a certain amount of accuracy. However, we have also shown that citations are incomplete assessments and that one cannot assume that a high number of citations correlate with a high level of usefulness. Only when one knows that a rarely cited p aper has been widely read is it possible to say – strictly spe aking – that it was obviously of little use for further research. Using a comparison with 'like' data, we have tried to determine that cited reference anal ysis allows a more meaningful anal ysis of bibliometric data than times-cited analysis. 11 For example a cited reference anal ysis for one scientist could provide information about the extent to which he or she refers to older or more recent publi cations, which journals he or she reads most (and then cites), the research is based, which theoretical appro aches to research he or she prefers and the qualit y o f the cited paper. This latter could provide information about whether the s cientist is able to identif y high-qualit y publications and incl ude them in his or her own work. An evaluation of research which does not use citations is unthinkable in the natural sciences nowadays (Bornmann et al., in press ; Bornmann & Marx, 2013c). The great popula rity of indicators such as the h index and the journal impact factor (although not the most appropriate bibliometric indicators) are stron g evidence of t his. The importance of cit ations for research evaluation purposes has been challenged neither by the exist ence of negative citations, self- citation s, o gift-citations (Bornmann & Daniel, 2008 ) and of the well-k nown phenomenon 'obliteration by incorporation' (McCain, 2011 ; Merton, 1968), nor b y recent discussions on the correlations between citation patterns and the st atus of findings in science as 'creative,' 'mainstream,' 'contested,' and 'ignored' (Heinze, 2013). Even if many studies suggest that not only th e content of scientific work , but also other, i n part non-scientific factors play a role in citing behaviour, one should not conclude that ci tations are an in appropriate indicator of impact of research. According to van Raan (2005) the process of citation certainly does not provide “ an 'ideal' monitor on scientific performance. This is particularly the case at a statisticall y low aggregation level, e.g. the individual researcher. There is, however, sufficient evidence that these reference motives are not so different or ‘ randoml y given ’ to such an ex tent that the phenomenon of citation would lose its 12 role as a reliable me asure of impact. Therefore, application of citation analysis to the entire work, the 'oeuvre' of a gr oup of researchers as a whole over a longer period of time (author’s emphasis), does yield in many situations a strong indicator of scientific performance ” (pp. 134-135). With this Brief Communication, it is not our intention to cast further doubt on the use of citation anal ysis for research evaluation; we m erely wish to posit – somewhat provocatively – the statement that given the limited significance of times -cited data, in many cases a cited reference analysis would make more sense. We would be pleased if putt ing this thesi s forward has stimulated lively discussion amongst bibliometricians. 13 References Bornmann, L . (2011). S cientific peer review. Annual Review of Information Science and Technology, 45 , 199-245. Bornmann, L., Bowman, B. F., Bauer, J ., Marx, W ., Schier, H., & Palzenberger, M. (in press). Standards for using bibliometrics in the evaluation of r esearch inst itutes. In B. C ronin & C. Sugimoto (Eds.), Next generation metrics . Cambridge, MA, USA: MIT Press. Bornmann, L., & Daniel, H.-D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64 (1), 45-80. doi: 10.1108/00220410810844150. Bornmann, L., de Mo ya-Anegón, F., & Le ydesdorff, L. (2010). Do s cientific advancements lean on the shoulders of giants? A bibliometric investigation of the Ortega hypothesis. PLoS ONE, 5 (10), e11344. Bornmann, L., & Marx , W. (2013a). How good is research reall y ? Measuring the citation impact of publi cations with percentiles increases co rrect assessments and fair comparisons. EMBO reports, 14 (3), 226-230. doi: 10.1038/embor.2013.9. Bornmann, L., & Marx , W. (2013b). The proposal of a b roadening of p erspective in evaluative bibliometrics by complementing the times cited with a cited re ference analysis. Journal of Informetrics, 7 (1), 84-88. doi: DOI 10.1016/j.joi.2012.09.003. Bornmann, L., & Marx, W. (2013c). Proposals of standards for the application o f scientometrics in the evaluation of individual researchers working in the natural sciences. Zeitschrift Fur Evaluation, 12 (1), 103-127. Cole, S. (1992). Making science. Between nature and society . Cambridge, MA, USA: Harvard University Press. de Bellis, N. (2009). Bibliometrics and citation analysis: from the Science Citation Index to Cybermetrics . Lanham, MD, USA: Scarecrow Press. Di Vaio, G., Waldenström, D., & Weisdorf, J . (2012). Citation success: evidence from economic history journal publications. Explorations in Economic History, 49 (1), 92- 104. doi: 10.1016/j.eeh.2011.10.002. Froghi, S., Ahmed, K., Finch, A., Fitzpatrick , J. M., Khan, M. S., & Dasgupta, P. (2012). Indicators for research performance evaluation: an overview. BJU International, 109 (3), 321-324. doi: 10.1111/j.1464-410X.2011.10856.x. Galton, F. (1907). Vox populi. Nature, 75 , 450-451. doi: Doi 10.1038/075450a0. Geim, A. K., & Kim, P. (2008). Carbon wonderland. Scientific American, 298 (4), 90-97. Geim, A. K., & Novosel ov, K. S . (2007). The rise of graphene. Nature Materials, 6 (3), 183- 191. doi: 10.1038/Nmat1849. Heinze, T. (2013). Creative accomplishments in science: definition, theoretical considerations, examples from science history, and bibliometric findings. Scientometrics, 95 (3), 927-940. doi: DOI 10.1007/s11192-012-0848-9. Herron, D. M. (2012). Is expert peer review obsol ete? A model suggests th at post -publication reader review ma y exceed the accuracy of traditional peer review. Surgical Endoscopy and Other Interventional Techniques, 26 (8), 2275-2280. doi: 10.1007/s00464-012- 2171-1. Kessler, M. M. (1963). Bibliographic coupling between sci entific papers. American Documentation, 14 (1), 10-25. Kreiman, G., & Mauns ell, J. H. R. (2011). Nine criteria for a measure of scientific output. Frontiers in Computational Neuroscience, 5 . doi: 10.3389/fncom.2011.00048. Kuhn, T. (1970). Logic of discover y or ps ychology of rese arch? In I. Lakatos & A. Mus grave (Eds.), Criticism and the growth of knowledge (pp. 1-23). London, UK: Cambridge University Press. 14 Leydesdorff, L. (1994). The generation of aggregated journ al-journal citation maps on th e basis of the CD-ROM version of the Science C itation Index. Scientometrics, 31 (1), 59-84. doi: 10.1007/bf02018102. Martin, B. R., & Irvine, J . (1983). Assessing b asic research - some partial indicators of scientific progress in radio astronomy. Research Policy, 12 (2), 61-90. McCain, K. W. (2011). Epon ymy and obliteration by incorporation: the case of the “Nash Equilibrium”. Journal of the American Society for Information Science and Technology, 62 (7), 1412-1424. doi: 10.1002/asi.21536. Merton, R. K. (1968). Social theory and social structure . New York, NY, USA: Free Press. Merton, R. K. (1973). T he sociology of science: theoretical and empirical investigations . Chicago, IL, USA: University of Chicago Press. National Science Board. (2012). Science and engineering indicators 2012 . Arlington, VA, USA: National Science Foundation (NSB 12-01). Small, H. G. (1978). Cited documents as concept symbols. Social Studies of Science, 8 (3), 327-340. Smith, A., & E yse nck, M . (2002). The correlation between RAE ratings and citation counts in psychology . London: Department of Ps ychology , Ro yal Hollowa y, University of London, UK. Surowiecki, J. (2004). The wisdom of crowds . New York, NY, USA: Random House. van Raan, A. F. J . (1996) . Advanced bibliometric m ethods as quantitative core of peer review based evaluation and foresight exerc ises. S cientometrics, 36 (3), 397-420. van Raan, A. F. J. (2005). Fatal attraction: con ceptual and methodolo gical problems in the ranking of universities by bibl iometric methods. Scientometrics, 62 (1), 133-143. 15 Table 1: Times-cited and cited refere nce anal ysis using the e xample graphene re search in 2010. Times cited Most-cited paper fr om graphene research in 2010 Cited reference counts Papers most- cited in graphene resea rch in 2010 1054 BAE S, NA T NANOTECHNOL 2010 V5 P 574 1272 NOVOSELOV K S, SCIENCE 2004 V306 P666 942 DREYER D R, CHEM SOC REV 2010 V3 9 P228 928 GEIM A K, NAT MATER 2007 V6 P183 637 ZHU Y W, ADV MATER 2010 V22 P390 6 708 NOVOSELOV K S, NATURE 2005 V4 38 P197 633 ALLEN M J, CHEM REV 2010 V110 P132 640 CASTRONETO A H , REV MOD PHYS 2009 V 81 P109 584 LIN Y M, SCI ENCE 2010 V327 P662 629 ZHANG Y B, NATURE 2005 V438 P201 579 BONACCORSO F, NAT PHOTONICS 2010 V4 P6 11 383 STANKOVI CH S, NATURE 2006 V442 P28 2 571 SCHWI ERZ F, NAT NANOTECHN OL 2010 V5 P487 351 BERGER C , SCIENCE 2006 V312 P1191 450 DEAN C R , NAT NANOTECHNO L 2010 V5 P722 342 HUMMERS W S, J AM CHEM SOC 1958 V80 P1339 401 QU L T, ACS NA NO 2010 V4 P1321 311 KIM K S, NATURE 2 009 V457 P706 374 CAI J M, NA TURE 2010 V466 P470 309 FERRARI A C, PHYS REV LETT 2006 V 97 P187401 371 ZHANG H, ACS NANO 2010 V4 P380 307 GEIM A K, SCI ENCE 2009 V324 P1530 369 WU Z S, AC S NANO 2010 V4 P3187 294 STANKOVI CH S, CARBON 2007 V4 5 P1558 Note. Source: SCI from STN International