Accueil Revues Revue Numéro Article

M@n@gement

2013/5 (Vol. 16)

  • Pages : 238
  • DOI : 10.3917/mana.165.0697
  • Éditeur : AIMS

ALERTES EMAIL - REVUE M@n@gement

Votre alerte a bien été prise en compte.

Vous recevrez un email à chaque nouvelle parution d'un numéro de cette revue.

Fermer

Article précédent Pages 697 - 706 Article suivant

INTRODUCTION

1

Thompson Scientific is a database company that owns and publishes the Institute for Scientific Information (ISI) Web of Knowledge, which includes the Science Citation Index, the Social Science Citation Index, and the Journal Citation Reports. Central to the Journal Citation Reports are journal Impact Factors, which Thompson/ISI (1994) describes as “a systematic and objective means to critically evaluate the world’s leading journals.” The Impact Factor was devised in the 1960s by Eugene Garfield as a way to measure journal usage based on the mean number of citations per article within a specific period of time. A journal’s Impact Factor is calculated by counting the number of current-year citations to articles published by the journal during the preceding two years and dividing this count by the number of articles the journal published in those two years. More recently, the ISI introduced a five-year Impact Factor (i.e., current-year citations to articles published by the journal during the preceding five years divided by the number of articles published in those years) to account for differences in the diffusion and obsolescence of ideas across fields.

2

Garfield’s original idea was to sort journals by citation rates to aid in determining which to include in library collections (or indexes). Over the last decade, however, increasing electronic availability, along with aggressive marketing by Thomson Scientific, which acquired the ISI in 1992, has transformed the Impact Factor from a sorting device into a definitive quantitative rating of the quality of journals, of particular articles appearing in them and, by corollary, of the academics writing those articles. The journal Impact Factor is now in widespread use to evaluate researchers, serving central functions in academic hiring, peer review, and grant decisions—uses for which it was never intended and which Garfield (2006) himself has called misleading and inappropriate [1][1] In response, researchers increasingly emphasize publishing.... As a result, the Impact Factors of journals in which a researcher tends to publish are increasingly central to evaluations of his/her scholarly achievements. Indeed, the tendency has increasingly been to ascribe the Impact Factor of a journal to each article published within it. The veracity of such attributions rests on the assumption that a journal’s Impact Factor is representative of its articles. For this to be true, the citedness of a journal’s articles must follow a Gaussian distribution, with a narrow variance around the mean—that is, around its Impact Factor. It is well-known, however, that the distribution of citations to a journal’s articles is highly skewed, with few articles near the mean. The skewness of article citedness is also problematic for Impact Factors as an index of journal quality because mean journal citedness is disproportionately influenced by a small number of highly cited articles. A small minority of articles, unrepresentative of the journal’s publications, may thus be decisive in determining journal Impact Factors and resultant journal quality rankings (Baum, 2011).

3

To address this problem, I propose an adjustment for journal Impact Factors to account for the distribution of citations a journal receives. The adjustment is derived from two recent extensions to the h-index (Hirsch, 2005) [2][2] The h-index is the number of articles, h, that receive...: e2, which captures excess citations beyond the h2 citations received by h-core articles (Zhang, 2009), and t2, which captures surplus citations received by h-tail articles that fall below the h-core (Ye and Rousseau, 2010). A journal’s citation distribution is gauged by the “excess-tail” ratio, e2/ t2, which indicates the relative citedness of the few highly cited articles comprising the journal’s h-core and the many infrequently cited articles falling in its h-tail. The excess- tail ratio is used to compute as IF x e2/ t2, where IF is the raw journal Impact Factor and is the adjusted journal Impact Factor. Using the excess- tail ratio to adjust journal Impact Factors maintains the advantage of having a single index with which to evaluate journals, while incorporating important information on journal citation distributions.

4

I compute excess-tail ratios and for a sample of 25 management and economics journals. I find e2/ t2 < 1 (often << 1) for 23 of the sample journals. These results are consistent with an overstatement of raw Impact Factors, attributable to their sensitivity to a few highly cited articles. I also find that is more stable than raw Impact Factors across standard two- and five-year time horizons.

GAUGING JOURNAL CITATION DISTRIBUTION WITH THE EXCESS-TAIL RATIO

5

I gauge a journal’s citation distribution using the ratio of, on the one hand, excess citations to the few highly cited articles in the journal’s h-core to, on the other, surplus citations to the many infrequently-cited articles in the journal’s h-tail. The h-index divides a journal’s articles into two groups: the first group is the h-core, each having at least h citations during the period under study, and the second is the h-tail, each having at most h-1 citations.

6

The h-index, h- core and h-tail can be applied to many source–citation relations over many time windows (Ye and Rousseau, 2010). If there are S source articles and C citations, by definition the h-core consists of h articles and the h-tail consists of S – h articles. The number of citations in the h-core, Ch, is a minimum of h2 but has no upper limit. Zhang (2009) recently defined e2, comprised of Ch – h2 citations, to distinguish “excess” citations ignored by the h-index. The number of citations in the h-tail, t2, ranges from 0 to (S - h) (h -1).

7

The relation between h-core and h-tail citations is illustrated in Figure 1, which represents the citedness of a journal’s articles assuming a continuous citation function. In the figure, citations to the journal’s h-core articles, Ch, are the sum of citations in the h2 and e2 areas. The t2 area represents the surplus citations received by the journal’s h-tail articles. The excess-tail ratio, e2/ t2, gauges a journal’s citation distribution based on the ratio of excess h-core to surplus h-tail citations. When e2/ t2 >1, citations tend to be excess citations to the few articles comprising a journal’s h-core articles. When e2/ t2 <1, citations tend to be surplus citations to the many articles comprising a journal’s h-tail. Thus, the larger the ratio, the greater the extent to which a journal’s citations reflect excess citations to its few most highly cited articles relative to surplus citations to its many infrequently cited articles [3][3] Substituting Ch/ t2 for e2/ t2 gives identical results;....

Figure 1 - Geometric representation of e2, t2, h, and h2

COMPARISON OF MANAGEMENT AND ECONOMICS JOURNALS

8

Publication data for the period 2000–2010, citation data for the period 2000– 2011, and two- and five-year journal Impact Factors for 2010 were collected from Thomson Reuters/ISI Web of Knowledge for 25 management and economics journals [4][4] Using publication data for the period 2006–2010 and.... Based on these data, I computed h, e2, and t2, as well as excess-tail ratios, and the adjusted journal Impact Factors, IF. Figure 2 ) provides an illustrative exemplar of the calculations for the Quarterly Journal of Economics, and Table 1 summarizes h, h2, e2, t2 and the excess-tail ratios for all journals [5][5] Articles in a journal’s h-tail may receive no citations,....

Figure 2 - Quarterly Journal of Economics citations, 2000-2010

Note: Ninety of the 457 articles published in the Quarterly Journal of Economics during the period 2000-2010 received at least 90 citations, giving h = 90 and h2 = 8,100. The total number of citations received by these 90 “h-core” articles is, however, 16,021, yielding e2 = 16,021 – 8,100 = 7,921. The total number of citations to the remaining 367 “h-tail” articles (each receiving < 90 citations) is 11,025, which gives t2. Accordingly, e2/t2 = 0.718, indicating an inflated Impact Factor.

Table 1 - Citation statistics and ratios for selected management and economics journals
9

With two exceptions (Administrative Science Quarterly and Journal of Economic Literature), surplus h-tail citations exceed excess h-core citations, resulting in an excess-tail below 1 and for many journals << 1. Among the sample journals, modal citations thus tend to be citations to infrequently cited h-tail articles. As a result, a journal’s IF is generally smaller (and often ) significantly so) than its raw Impact Factor (Table 2). This result is consistent with raw journal Impact Factors being inflated by a few highly cited articles in a journal’s h-core, when citations are more typically to one of a larger number of infrequently cited articles in the journal’s h-tail [6][6] The excess-tail ratio, e2/t2, can also be used to adjust....

Table 2 - Raw and adjusted impact factors and rankings for selected management and economics journals

)

Note: IF = IF x e2/ t2

10

After making these adjustments the relative ranks of the journals shift, and sometimes substantially, as illustrated in Table 2. The lower a journal is ranked based on relative to its raw Impact Factor, the greater the initial overstatement of its impact due to the sensitivity of its raw Impact Factor to a few highly cited h-core articles relative to the mass of infrequently cited h-tail articles. As illustrated in Figure 3, the correlation between two- and five-year rankings is substantially larger for (r = 0.98) than for raw Impact Factors (r = 0.88). is thus more consistent over different citation time horizons, particularly among higher-ranking journals.

Figure 3 - Correlation between two- and five-year raw and adjusted journal Impact Factors

CONCLUSION

11

As a measure of research quality, journal Impact Factors are problematic. The tendency to attach the same value to each article published in a given journal masks extreme variability in article citedness, and permits a journal’s many infrequently cited articles—and the journal itself—to free-ride on the journal’s few highly cited articles, which are principal in determining the journal’s Impact Factor (Baum, 2011). I propose a correction for this problem whereby a journal’s raw Impact Factor is adjusted to account for its citation distribution, which is gauged by the ratio of its excess h-core to surplus h-tail citations. This excess-tail ratio captures the extent to which the journal’s citations are centered on the more or less frequently cited articles, and thus the more or less influential articles it publishes.

12

I employ the excess-tail ratio to recalibrate Impact Factors for 25 selected journals in management and economics. The excess-tail ratio is less than 1 (and often << 1) for all but two sample journals. This is consistent with an overstatement of raw journal Impact Factors resulting from their sensitivity to small numbers of highly cited articles, and an inability to discriminate the shapes of the underlying journal citation distributions. Thus, while journal Impact Factors in management and economics are driven by citations to the journals’ small number of influential h-core articles, more typically their citations are to one of the large number of infrequently cited h-tail articles they publish. Moreover, adjusted Impact Factors (IF) produce rankings that differ (often substantially) from the raw rankings, and are more consistent across two- and five-year time horizons.

13

The excess-tail ratio and IF thus appear to provide useful complements to journal Impact Factors in assessing journal impact and quality, particularly given the increasing use of journal Impact Factors in the evaluation of individual scholarly output. Journal Impact Factors adjusted by these ratios carry additional information derived from journal citation distributions. As a result, IF would appear to afford a more accurate single-number metric for the evaluation of journals and the authors who publish in them.


REFERENCES

  • · Baum, J. A. C. (2011). Free-riding on power laws: Questioning the validity of the Impact Factor as a measure of research quality in organization studies. Organization, 18 (4), 449-466.
  • · Garfield, E. (2006). The history and meaning of the journal impact factor. Journal of the American Medical Association, 295 (1), 90-93.
  • · Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102 (46), 16569-16572.
  • · Institute for Scientific Information (1994). The ISI Impact Factor. Thomson Scientific. Retrieved from: http:// thomsonreuters.com/products_ services/science/free/essays/impact_ factor/
  • · Monastersky, R. (2005, October 14). The number that is devouring science. Chronicle of Higher Education.
  • · Reedijk J., & Moed, H. F. (2008). Is the impact of journal impact factors decreasing. Journal of Documentation, 64 (2), 183-192.
  • · Rousseau, R. (2006). New developments related to the Hirsch index. Science Focus, 1, 23-25 (Chinese). English translation retrieved from: http://eprints.rclis.org/6376/
  • · Wilhite, A. W., & Fong, E. A. (2012). Coercive citation in academic publishing. Science, 335 (6068), 542-543.
  • · Ye, F. Y., & Rousseau, R. (2010). Probing the h-core: An investigation of the tail-core ratio for rank distributions. Scientometrics, 84, 431-439. doi: 10.1007/s11192-009-0099-6
  • · Zhang, C. T. (2009). The e-index, complementing the h-index for excess citations. PLoS ONE, 4 (5), e5429.
  • · Zhang, C. T. (2013). The h’-Index, effectively improving the h-Index based on the citation distribution. PLoS ONE, 8 (4), e59912.

Notes

[1]

In response, researchers increasingly emphasize publishing in journals with high Impact Factors rather than journals that might be more appropriate for their research, and may alter the kind of studies conducted to accommodate the predilections of such journals. This emphasis on Impact Factors has led some observers to comment that what a researcher contributes to our understanding is in danger of becoming less important than where it is published (Monastersky, 2005). This attention has also encouraged coercive editorial strategies to manipulate the system (Reedijk and Moed, 2008; Wilhite and Fong, 2012), as well as publisher policies to combat them (see, e.g., http://editorsupdate.elsevier.com/2012/06/ impact-factor-ethics-for-editors/).

[2]

The h-index is the number of articles, h, that receive at least h citations; thus, the h-index is 25 for a journal that published 25 articles receiving at least 25 citations.

[3]

Substituting Ch/ t2 for e2/ t2 gives identical results; the correlation between the two is r = .996 for the sample of journals examined below.

[4]

Using publication data for the period 2006–2010 and citation data for the period 2006–2011 results in lower values of e2/t2 for all journals, but does not otherwise alter the main findings or their implications.

[5]

Articles in a journal’s h-tail may receive no citations, and so contribute nothing to t2. When some h-tail articles go uncited, the excess-tail ratio may therefore overstate the “quality” of a journal’s citations. To correct for this, t2 can be substituted with the number of “reverse tail” citations rt2; that is, the difference in the actual number of h-tail article citations and the number of h-tail citations if all h-tail articles received h citations. More formally, rt2 = t2 – (S - h) (h - 1), which ranges from 0 to (S - h) (h - 1). Because, for the 25 sample journals, the proportion of uncited h-tail articles is relatively small (mean =.058; min = .006; max = .171), the correlation between e2/t2 and e2/rt2 is 0.949, and the impact of this substitution on adjusted Impact Factors is negligible. Nevertheless, if the proportion of uncited h-tail articles is large, this substitution may be material.

[6]

The excess-tail ratio, e2/t2, can also be used to adjust the h-index itself, improving its ability to discriminate the shapes of citation distributions similarly (Zhang 2013).

Résumé

English

Despite their widespread adoption, journal Impact Factors suffer well-known drawbacks that limit their usefulness in accurately and fairly assessing scientific quality. Among these is the extreme variance and skewness in the citations to articles published by a given journal, which results in their sensitivity to a few highly cited articles, and enables many infrequently cited articles to “free-ride” on citations to these “skewed few.” To address this problem, I adjust journal Impact Factors according to the relative citedness of the few highly cited articles in a journal’s h-core (i.e., the h articles that receive at least h citations) and the many infrequently cited articles in its h-tail (i.e., those that receive fewer than h citations). I gauge the skew of a journal’s citation distribution by e2/ t2, the excess-tail ratio where e2 captures excess citations above the h2 citations received by a few highly cited h-core articles and t2 captures surplus citations received by the many infrequently cited h-tail articles that fall below the h-core. I employ e2/ t2 to adjust raw Impact Factors for 25 selected management and economics journals. The adjusted scores, , discriminate Impact Factors based on the shapes of journal citation distributions, leading to more accurate evaluation. I find e2/ t2 < 1 (often << 1) for 23 of these journals to be consistent with an overstatement of their quality resulting from the sensitivity of Impact Factors to? a few highly cited articles. Adjusted Impact Factors also yield distinctive and more consistent journal rankings over standard two-year and five-year time horizons. I conclude that the “excess-tail” ratio and are a useful complement to journal Impact Factors, particularly given their increasing use in the evaluation of individual scholarly output.

Keywords

  • h-Index
  • e-Index
  • h-core
  • h-tail
  • excess-tail ratio
  • journal Impact Factor

Plan de l'article

  1. INTRODUCTION
  2. GAUGING JOURNAL CITATION DISTRIBUTION WITH THE EXCESS-TAIL RATIO
  3. COMPARISON OF MANAGEMENT AND ECONOMICS JOURNALS
  4. CONCLUSION

Pour citer cet article

Baum Joel A.C., « The excess-tail ratio: correcting journal impact factors for citation distributions », M@n@gement, 5/2013 (Vol. 16), p. 697-706.

URL : http://www.cairn.info/revue-management-2013-5-page-697.htm
DOI : 10.3917/mana.165.0697


Article précédent Pages 697 - 706 Article suivant
© 2010-2014 Cairn.info
back to top
Feedback