Remember me
REVIEW ARTICLE
Year : 2022 | Volume
: 17
| Issue : 6 | Page : 319-327
Publication metrics: Types, utility, and caveats
Durga Prasanna Misra
Department of Clinical Immunology and Rheumatology, Sanjay Gandhi Postgraduate Institute of Medical Sciences (SGPGIMS), Lucknow-226014, India
Correspondence Address:
Dr. Durga Prasanna Misra
Department of Clinical Immunology and Rheumatology, Sanjay Gandhi Postgraduate Institute of Medical Sciences (SGPGIMS), Lucknow-226014
India
Source of Support: None, Conflict of Interest: None
CheckDOI: 10.4103/0973-3698.364674
Publication metrics enable the assessment of the performance of citations or utilization of published work. Journal-level metrics depend on the database whose citations are analyzed. Publication metrics from the Web of Science include the widely-used journal impact factor (JIF) and other indices such as Journal Citation Impact, Eigenfactor, normalized Eigenfactor, and Article Influence Score. Metrics from Scopus include the CiteScore, Source Normalized Impact Factor, and SCImago Journal Rank. Author-level metrics such as total citations, h-index, i10-index, and g-index inevitably increase with time and can be inflated by self-citations. Article-level metrics such as total citations and online attention scores derived from Scopus (PlumX Metrics) or Altmetric indicate the extent of utilization and discussion in scientific circles of a particular article. Publication metrics are useful to provide a bird's eye view of how well an individual journal or article has been cited or used. They do not necessarily reflect article quality. As an example, some of the articles with the highest Altmetric Attention Scores are actually retracted publications that attained high scores due to the negative debate that they elicited. Journal-level metrics such as the JIF are fluid and prone to historical fluctuations from year to year, most recently observed by increases in the JIF of journals which published a lot of coronavirus disease 19-related content. Research assessment of individual scientists often misuses publication metrics such as the JIF, when they should instead rely on a critical appraisal of research articles by assessors.
Keywords: Citation analysis, CiteScore, h-index, i10-index, journal impact factor
After an article is published in a scientific journal, it could be read, discussed, or cited. Publication metrics refer to the analysis of data related to the utilization of articles after publication. Publication metrics could relate to the journal where articles are published, to the authors of scientific articles, or to the articles per se. A lot of the data related to publication metrics is derived from the bibliographic databases where such scientific articles are indexed.
Analysis of publication metrics has been historically used for research evaluation. As shall hopefully become evident after further reading through this review, publication metrics have their own limitations. The research output of an individual researcher cannot be completely understood by just a single figure such as the h-index or others. Over the past decade, numerous international organizations (the Leiden manifesto from European researchers),[1] the San Francisco Declaration on Research Assessment (SFDORA) from the United States of America,[2] and the Metric Tide from the United Kingdom[3] have highlighted these limitations of simply using publication metrics for research assessment. These declarations recommend a holistic assessment of researcher efforts in universities for recruitment or promotion, including their relevance in the field of the specific researcher and their wider relevance, their relevance to the research mission of the university or hospital, and the actual assessment of the quality of the research work rather than simply noting where it was published or how many times it has been cited. In this context, we shall discuss various journal-level, author-level, and article-level metrics [summarized in [Figure 1]] as well as critically analyze the utility and pitfalls of publication metrics.
Figure 1: Bird's eye view of commonly-used journal-level, author-level, and article-level metrics Journal-Level Metrics Journal-level metrics are derived from citation and usage data of scientific journals in the databases that they are indexed in. For articles in medical publishing, most of the journal-level metrics are derived from either the Web of Science (WoS) or Scopus.
Journal-Level Metrics from the Web of ScienceThe WoS is a large bibliographic database. It is operated by Clarivate Analytics (formerly by Thomson Reuters). The WoS core collection encompasses numerous databases, of which those of greater relevance to the medical academic publishing community are the Science Citation Index Expanded (SCIE) and the Emerging Sources Citation Index (ESCI).[4] The annual Journal Citation Report (JCR) is based on data from the WoS and provides numerous useful journal-level metrics including the widely used journal impact factor (JIF).[5]
Journal impact factor
Devised by Eugene Garfield, the JIF is based on an analysis of citations to articles published in the WoS. For a given year, journals indexed in the SCIE and the Social Sciences Citation Index (SSCI) are provided with a JIF. This is calculated based on the number of citations to articles published in the past 2 years (2-year JIF) or 5 years (5-year JIF), divided by the number of citable items published in the past 2 years/5 years in such journals. Research articles and review articles are considered citable items. Letters to the editor, editorials, case reports, and other types of articles are not considered citable items. Keeping in mind that citations take time to gather after an article has been published, the 5-year JIF might provide a more accurate estimation of the average citability of journal articles in a particular journal.[5],[6] However, the 2-year JIF is more commonly publicized and discussed.
Recent changes in the methodology used to calculate the JIF are important to understand. In the 2021 release of the JCR, the JIF calculation took into account the date that articles were first published online (instead of the date of print publication).[7] Previously, journals were upgraded from the ESCI to the SCIE if they performed well enough. Although no cutoffs had been proposed for how well journals had to perform, it is widely believed that journals would be upgraded if their prospective JIF would rank in the first or second quartile of the journals in their field. A recent press release from Clarivate analytics mentioned that from the next release of the JCR in 2023, all journals indexed in the WoS (not just those in SCIE and SSCI) shall be provided with a JIF. Future iterations of the JCR shall include JIFs up to one decimal place alone, to avoid unwarranted attention to minuscule differences in the JIF between journals, in an attempt to promote a more holistic assessment of journal-level metrics. Furthermore, journals with a JIF less than 0.1 shall simply have this mentioned as <0.1 rather than providing an exact value.[8] There is no upper limit for the JIF. As an example, the JIF for the New England Journal of Medicine is 176.071,[9] and for the Lancet Rheumatology is 35.482.[10]
The JIF is widely used as a measure of the prestige of a journal. Publishing in a journal with a “high” impact factor is considered prestigious. This is fallacious because publishing in a “high” impact factor journal does not reflect the quality of an individual article published in the journal, but rather reflects only the average number of citations an article garners in the next 2 years/5 years in that particular journal. Certain types of articles such as review articles are more likely to get cited than original articles, therefore, journals publishing only review articles or a greater proportion of review articles are likely to have a higher JIF. As discussed subsequently, the JIF is prone to significant instabilities over time.[11] Authors should be aware of the pitfalls associated with the JIF and should not mistake this measure as the ultimate expression of journal quality.
Journal citation impact
First published in the 2021 release of the JCR, the Journal Citation Impact (JCI) attempts to normalize the citation performance of a particular journal indexed in the WoS based on the number of citations, the type of article, and the category of the article (i.e., the type of article, subject area, and how long it has been published). Based on these calculations, each research or review article is provided a Category-Normalized Citation Index which is a ratio of actual citations to expected citations based on the article category. The JCI for a particular journal is calculated taking into account the Category-Normalized Citation Index for all articles published in a journal. A JCI of more than 1 indicates that the journal is performing better than the average journal of a similar category. The JCI enables comparisons across journals which do or do not have a JIF.[12]
Eigenfactor, normalized Eigenfactor, and article influence score
The Eigenfactor takes into account not only the number of citations but also the quality of the journals from where such citations are garnered. Calculating the Eigenfactor is based on the assumption that more prestigious journals would cite other similarly prestigious journals. Thus, the Eigenfactor measures overall journal influence. Citations to articles published in the same journal are not considered while calculating the Eigenfactor. Overall, the Eigenfactor scores from all journals in the JCR would total 100. Based on the way the Eigenfactor is calculated, journals publishing a larger number of articles shall inherently have a greater Eigenfactor. The Article Influence Score (AIS) normalizes the Eigenfactor for the number of articles published in the journal by averaging the Eigenfactor with the number of articles in that particular journal over the past 5 years. An AIS greater than or lesser than one indicates that the influence of a particular article is above average or below average, respectively. The normalized Eigenfactor adjusts the Eigenfactor in relation to the total number of journals published in the WoS.[13]
Immediacy Index
The Immediacy Index takes into consideration the citations to a particular article in the year that has been published. It is expected that more influential articles shall be cited to a greater extent earlier in their life course, therefore, shall have a greater Immediacy Index.[14]
Journal rank
The journal rank is based on the quartile (Q1 to Q4) the journal falls within based on its JIF.
Journal-Level Metrics from the ScopusScopus is a bibliographic database from Elsevier. It has wider coverage than the WoS.[15] Scopus includes >95% of the journals that are available in MEDLINE.[16] Analysis of citations from Scopus at the journal level provides the CiteScore, Source-Normalized Impact Factor, and the SCImago Journal Rank (SJR), all of which consider only peer-reviewed articles for their calculation.
CiteScore
The CiteScore provides the number of citations to articles published in Scopus-indexed journals over the past 4 years averaged out by dividing the total number of citable items in these journals in the past 4 years. Citable items include research and review articles, conference abstracts, data papers, and textbook chapters. The methodology for calculating the CiteScore changed in 2020 to include 4 years as the citation window instead of the previous window of 1 year and to define the types of articles that would be included in the denominator as well as the numerator, i.e., citations to which would be considered for calculating the CiteScore.[17] There is no upper limit for the CiteScore. As an example, the CiteScore for the New England Journal of Medicine is 110.5,[18] and for the Lancet Rheumatology is 23.0.[10][Table 1] compares the JIF and the CiteScore.
Table 1: Comparison between the journal impact factor and the cite scoreSource-Normalized Impact Factor
Citations in different fields or different subject areas are likely to follow distinct patterns. Articles in some fields might be cited more frequently than in other fields. The Source-Normalized Impact Factor (SNIP) averages the mean number of citations to articles in a journal with the citations received by journals in that particular field.[19]
SCImago journal rank
Conceptually similar to the Eigenfactor, the SJR takes into account the prestige of journals providing citations to a particular journal. The SJR normalizes average citations to articles from a journal to the rank of the journals (based on their SJR) citing such articles.[20] A ranking of quartiles of journals (Q1 to Q4) for journals in Scopus overall as well as those in a specific area of knowledge based on their SJR is also available.[21]
Other journal-level metrics
Most of the reputed journals today provide authors an indication of the times to first decision (including desk rejections) and time to publication (from submission or from acceptance). The individual journal articles also indicate this information through dates of submission, revision, and acceptance. Furthermore, journals publish acceptance rates on their websites (updating this information from time to time). These figures are useful for prospective authors, who usually favor quick turnaround times for decisions or publications.
Author-Level MetricsAuthor-level metrics are based on the analysis of publications or citations for a particular author. The number of citations to one's work inevitably increases with time. Therefore, a highly productive and influential early career researcher will inevitably have a lesser number of citations than a senior, established researcher in the field. The simplest author-level metrics are the number of publications or the number of total citations. A total number of publications provide little meaningful information because it does not account for the quality of publications. If a large proportion of published articles by a particular author are letters to the editor or case reports, then they do not have the same importance as another author with a similar number of publications most of whose articles are research or review articles. Similarly, the total number of citations to a particular author's work cannot distinguish between self-citations and nonself-citations (i.e., citations to one's work by others). An excessive number of self-citations can apparently inflate the total number of citations.[22] A rule of thumb is that self-citations should not exceed more than 15–20% of the citations in a particular article.
Other author-level publication metrics are the h-index, i10-index, and g-index. The h-index was proposed by George Hirsch. It is calculated as the largest number of publications each having received at least the same number of citations. For example, an h-index of 35 means that the particular author has published at least 35 articles which have been cited at least 35 times each.[22],[23],[24] The i10-index indicates the number of articles published by a researcher that have garnered at least ten citations each.[22],[24] Iterations of this concept can be used to calculate i20, i100, i1000, or similar indices.[25] The g-index can be calculated as the largest number of articles whose total citations account for the g-squared number of citations.[24] For example, an author with a g-index of 43 has 43 publications which account for 43 squared (i.e., 1849) citations. Common databases on which the h-index, i10-index, and g-index are calculated are Scopus, WoS, or Google Scholar. Since the number of citations varies with the database used, calculations for the h-index, i10-index, and g-index vary depending on the database used to calculate these metrics.[26] While Google Scholar has the advantage of being free to access, it also includes citations to work in newspapers, weblogs, and other sources. Studies have indicated the inaccuracy of citations on Google Scholar as this uses an automated algorithm to track citations.[27] In this context, it is important to reiterate that the database used for calculating an author-level metric as well as the date on which such a database was used for this purpose should always be mentioned.[11] In general, the i10-index is only available from Google Scholar, whereas, the h-index can be calculated based on Scopus, WoS, or Google Scholar data. The h-index, i10-index, and g-index can also be calculated for individual journals, although this is done less frequently.
Inappropriate self-citations can inevitably inflate all author-level metrics including the h-index, i10-index, and g-index.[22] The impact of self-citations on author-level metrics can be difficult to gauge. Few platforms such as ResearchGate provide values of h-index both including and excluding self-citations. Commercially available databases such as Scopus and WoS should consider providing similar distinctions in author-level metrics including and excluding self-citations.
Disambiguation of author names is important to avoid missing citations related to an author.[22],[28] By the use of unique identifiers such as the Open Researcher and Contributor ID (ORCID), iterations of an author's name with slightly different spellings can be appropriately linked to authors.[22],[28] At present, Scopus and WoS provide a facility to link ORCID with the Scopus Author identifier or with the researcher identifier on the WoS. Some funding agencies also mandate the linking of prospective grantees to ORCID, although most Indian funding agencies do not yet ask for this.[29]
It is also important to differentiate author profiles from author-level metrics. Author profiles such as those available on ResearchGate, Google Scholar, Academia.edu, and the National Center for Biotechnology Information, National Institutes of Health (through the ScienCV function) help the author to consolidate their research output for the ease of viewing and sharing. Some of these author profiles such as ResearchGate and Google Scholar provide author-level metrics as well.[30] Author profiles are also available through the researcher identifier on the WoS and with the Scopus Author identifier.[31]
Article-Level MetricsThe number of downloads or clicks on full-text links for a particular article is a useful measure of its utilization in the research community. Quite a few journal websites routinely provide this information. Another measure that could reflect the quality or relevance of an article is the number of citations garnered by the article. Needless to say, these increase over time. Self-citations (i.e., citations to an article by its own authors) should be done only when appropriate. Scopus provides PlumX Metrics for each article in its database. Another commonly used metric is the Altmetric. Since English is the most commonly used language for scientific communication, it is reasonable to suppose that articles published in English (as opposed to those published in other languages) shall receive more clicks, downloads, and citations.
PlumX MetricsPlumX Metrics provides a Plum Print which diagrammatically represents five different aspects of the usage of a particular article in distinct colors. The green blob indicates usage (clicks, downloads, and the like). The purple blob indicates captures, namely, the number of times an article has been bookmarked or earmarked by readers for future use. The yellow blob indicates mentions, i.e., the number of times an article has been mentioned in newspaper articles, blog spots, and the like. The blue blob indicates mentions of the article on social media channels such as Twitter and Facebook. The red blob indicates the number of citations garnered by the article on databases such as Scopus or PubMed Central or within policy documents or patents.[32] An example of the PlumX Metrics figure is provided for ease of understanding [Figure 2]a.[33],[34]
Figure 2: (a) Example of PlumX metric. The different colored blobs denote usage, captures, mentions (newspaper and social media separately), and citations. (b) Example of Altmetric Donut. The different colored rings denote Tweets, Facebook mentions, mentions in weblogs, citations in policy documents, and other different aspects related to usage Altmetric Donut and Score Altmetric provides the Altmetric Attention Score (AAS) and a colored donut to indicate different aspects of attention received by a particular article. Such attention might be from blog spots, social media channels, policy documents, Wikipedia, and other sources.[35] Not all sources of attention contribute equally to the AAS. For example, while Twitter posts are all counted toward the AAS, Facebook mentions on only specific public pages are accounted for.[36] An example of the Altmetric donut is provided for ease of understanding [Figure 2]b.[33],[37]
Utility of Publication MetricsJournal-level metrics
Metrics such as the JIF, JCI, Eigenfactor, AIS, and CiteScore indicate the average quality of the articles published in a journal (but not that of an individual article published in such journals). They enable direct comparisons of citations between two different journals. In this sense, they provide a benchmark for journals to guide their efforts toward further improvement of the quality of and attention received by articles published in them. The quartiles of journal rankings in the JCR and SJR databases also provide similar benchmarks for journals to strive for improvement.
Author-Level MetricsThese indicate the number of citations received by articles published by individual authors. It must be kept in mind that the number of citations (thereby, various author-level metrics) increases both with time and with better quality of articles. Therefore, author-level metrics cannot be compared between authors at different stages of their academic careers.
Article-Level MetricsThese are useful to indicate either the usefulness of articles for education and further research (thereby, indirectly their quality) or their ability to arouse interest among readers. Articles that induce discussion in scientific circles (such as those on controversial topics) might have better article-level metrics such as AAS or PlumX Metrics. Article-level metrics are also useful for bibliometric analyses to identify that are highly cited in a particular field. Such highly-cited articles might be essential reading for trainees or young scientists in a particular field to gain background or historical perspective on seminal research in that field.[26]
Caveats Associated with Publication MetricsJournal-level metrics
Different journal-level metrics are calculated using different methodologies and are specific for the particular database where they are calculated from. The JIF indicates only citations on the WoS just as the CiteScore only indicates Scopus data. Scientists and researcher appraisers alike often mistake indices such as the JIF as synonymous with the prestige of a journal.[38] Particularly, getting an article published in a “high impact factor” journal is considered “prestigious” by authors. This is fallacious because the JIF does not represent the quality of an individual article, rather only reflects the past citation patterns of the journal where the article is published. The quality of a said article can only be assessed by reading through it in depth, and indirectly by the number of citations that it garners in the future.[39] Reputed journals affiliated with professional societies with a long publishing record which are not indexed on the WoS will not have a JIF, yet, are considered prestigious in academic circles.[40] Journals which are listed on MEDLINE and available on a PubMed search need not have a JIF, as this requires a journal to be first indexed on the WoS. It is necessary to understand these limitations of the JIF to avoid misusing this journal-level metric.
Specific scientific fields such as humanities and literature inherently have low citation potential.[41],[42] Therefore, journals devoted to such areas inherently have numerically lower journal-level metrics (such as the JIF) unless normalized to the field.
Review articles and articles describing methodology are more likely to be cited. Therefore, journals publishing review articles either exclusively or predominantly are more likely to have a greater JIF or CiteScore.[42] The JIF is prone to considerable fluctuations from year to year. As an example, the JIF of the journal Acta Crystallographica Section A increased to 49.9 in 2009 from 2.0 in the preceding year.[43] This was because of a particular methods article that was published in the journal which had garnered more than 6000 citations in that particular calculation window for the JIF.[43],[44] As a consequence of this, this journal was ranked as number 2 in that particular JCR release. Inevitably, after the 2-year citation window had passed, the JIF for this journal came back to similar levels as in 2008.[43] In the 2022 release of the JCR, a number of journals have had considerable increases in the JIF when compared with last year.[45] A contributing factor toward such increases in the JIF is the fact that the 2022 JCR release is the first to consider citation data from 2020, the year when the coronavirus disease 19 (COVID-19) pandemic struck. As a consequence, journals publishing a considerable number of articles related to COVID-19 likely have had a considerable increase in their JIF this year. If this hypothesis holds true, the JIF of many journals should come back to the levels they were at before 2022 after 2 years. One of the reasons for increased citations to articles related to COVID-19 might be that most such articles were available free to read for all by publishers in view of the pandemic. Some studies but not others have suggested that articles published open access might receive more citations and thereby have better publication metrics.[46],[47]
Journal-level metrics such as the JIF can be gamed through citation cartels. Citation cartels could exist at the level of journals, namely, specific journals cite other journals preferentially and vice versa, thereby inflating their JIFs. This phenomenon is also called as citation stacking.[48] Authors could also preferentially cite specific authors' articles, thereby inflating the author-level metrics such as h-index or i10-index [Figure 3]. Needless to say, both practices are unethical. The JCR is known to suppress journal titles if these journals are suspected of indulging in citation stacking.[5]
Journal-level metrics cannot yet capture the quality of editorial and reviewer inputs, which are key components contributing toward the integrity and quality of the publication process. Editorial efforts to improve journals might result in an increase in journal-level metrics in the future but will not be reflected by current levels of such metrics.
Journal-level metrics are widely misused for the assessment of researchers during promotions and recruitment. Numerous consensus statements such as the SFDORA explicitly caution against this wrong practice. Despite the fact that such limitations of the JIF have been known for more than two decades, this metric continues to be abused for research assessment and appraisal of individual researchers.[42],[49] In part, this could be explained by the lack of awareness regarding the meaning of journal-level metrics. A fair research assessment requires an actual reading and critical appraisal of published research rather than simply noting the journal where it is published in and its “impact factor”. There is a need to bring in wider recognition of this phenomenon, particularly among senior academics who are responsible for such research assessments.[2]
Author-Level MetricsIt is essential to understand that data regarding citations varies from database to database. Therefore, author-level metrics which essential rely on an analysis of such citations will inevitably differ from one database to another. Therefore, while mentioning indices such as the h-index, the source database should be mentioned.[22] As discussed earlier, author-level metrics are not directly comparable between researchers at different stages of their academic careers because they are influenced both by the lifespan and the citability of such articles.
Article-Level MetricsArticle-level metrics such as the AAS and PlumX Metrics do not necessarily reflect eventual citations, rather only reflect the attention received by such articles on the world wide web. Mentions in print newspapers and other media not available on the Internet would not be picked up by such metrics. Mere attention does not necessarily equate with quality or citability. Increased attention might also reflect that an article is controversial and stimulates discussion on online channels. As an example, two articles related to COVID-19 which were retracted due to suspicions about fraudulent data have very high AAS in the thousands.[50],[51] Altmetric publishes the top 100 articles with the highest AAS for a particular year. In the list published for 2020, three of the top twenty articles on such a list were retracted articles which stimulated a considerable amount of online discussion. It is obvious that these particular articles were not of high quality despite their high AAS.[52][Table 2] summarizes the limitations and potential misuse of publication metrics.
ConclusionPublication metrics provide a bird's eye view of how well an individual journal or article has been utilized in terms of citations and usage. Journal-level metrics such as the JIF indicate only the average citation performance of any given article in a journal. However, the JIF and other journal-level metrics are widely misused for research assessment of individual authors, which is a wrong practice. Author-level publication metrics and citations to individual articles inevitably increase with time. Inappropriate self-citations might inflate such author-level and article-level metrics, therefore, publication metrics should never be taken as the only measure of the scientific validity of research or researchers.
Financial support and sponsorship
Nil.
Conflicts of interest
There are no conflicts of interest.
References
Comments (0)