Efficiency of research performance and the glass researcher

) doubt the validity of established size-independent indicators for measuring citation impact. In order to demonstrate the supposed lack of validity some examples are presented in their paper which are based on data from a comprehensive national database. The database contains standardized data on the Italian science system which includes not only output but also input indicators. Abramo and D'Angelo (in press) plead in favor of measuring scientific efficiency rather than performance, and refer to their proposal of the Fractional Scientific Strength (FSS) indicator (Abramo & D'Angelo, 2014). This is a composite indicator which considerswhen used on the university levelthe total salary of the research staff and the total number of publications which are weighted with citation impact. The argumentations and recommendations of Abramo and D'Angelo (in press) may sound reasonable from an economic standpoint, but are questionable in other respects. In the following, we take up some problematic and two interesting points.

(1) In the beginning of the paper, Abramo and D’Angelo (in press) question the validity of size-independent citation impact indicators for measuring research performance.

They argue that research performance can be validly measured only if input indicators are considered (e.g. citation impact per euro spent or FSS). There are several problems with this approach. (i) Scientometricians do not equate citation impact to research performance. The measurement of research performance includes several aspects other than impact alone. For example, the SCImago Institutions Ranking (http://www.scimagoir.com/ ) considers manifold sets of indicators which are categorized as research indicators (e.g. output and scientific talent pool), innovation indicators (e.g. technological impact), or web visibility indicators (e.g. website size). (ii) Abramo and D’Angelo (in press) confuse impact with efficiency and performance with productivity. They claim that size-independent indicators violate an axiom of production theory, but size-independent impact indicators were never intended to measure productivity. (iii) The plea of Abramo and D’Angelo (in press) for efficiency scores and against citation impact indicators is based on “basic economic reasoning”. The authors fail to substantiate their claim that one metric measures research performance validly and the others do not. The psychometric literature offers several ways to investigate validity which have already been applied in bibliometrics: the investigation of predictive validity studies the degree to which an indicator can predict other indicators of research performance which are measured in the future (Hirsch, 2007). For the investigation of the convergent validity, it is studied whether an indicator is correlated with other indicators which are specified to measure the same aspect (Bornmann, Mutz, & Daniel, 2008). However, the investigation of validity requires the comparison of different indicators measuring the same construct of performance.

Since citation indicators measure impact only and composite indicators (the FSS) consider several aspects of research performance; they do not claim to measure the same construct and cannot be fairly compared. In order to test the predictive or convergent validity of the FSS, Abramo and D’Angelo (in press) need at least one other indicator which measures research performance validly. Then, they can really judge the validity of the FSS.

(2) Abramo and D’Angelo (in press) argue for the use of a composite indicator to measure research performance. The indicator considers some input and output variables which are combined into a single number. Instead, we argue that research performance should not be measured by composite indicators, but by reporting the results of measuring different aspects of research performance separately. Research performance is a multi-dimensional phenomenon and different dimensions should be reported in an evaluation study. For example, Bornmann and Marx (2014) proposed using16 indicators (each of which is more or less important in a specific evaluation) to measure the productivity and citation impact of single researchers. As input measures one can use salaries, but also requested external funds or number of doctoral and post-doctoral researchers (in a professorship or institution). Moed and Halevi (2015) introduce the Multidimensional Research Assessment Matrix. The matrix can be used to decide which indicators are applied in a specific evaluation context. The basic assumption is that “the choice of metrics to be applied in a research assessment process depends on the unit of assessment, the research dimension to be assessed, and the purposes and policy context of the assessment” (Moed & Halevi, 2015). In order to study research performance in different contexts validly, we recommend following flexible schemes which can be adapted to a specific evaluation context rather than fixed formulas which

This content is AI-processed based on open access ArXiv data.