Research with Impact

Originally conceived to help libraries and researchers be selective in journal management, the influence of IF has grown and spread significantly, raising issues about the process of the calculation and its current uses and abuses.

According to Thomson Scientific, which computes journal IFs, the number is “a measure of the frequency with which the ‘average article’ in a journal has been cited in a particular year or period.” (See Figure 1)

In 2004, the fully automated system applied its algorithms to 27 million citations in 5,968 science journals and 1,712 social science journals. Originally, IF was intended as a practical quantitative tool to rank, evaluate, categorize, and compare journals worldwide. Now it is seen as a measure of the quality of a journal, its contents, and its authors. Recently, however, IFs have been used, or abused according to Dr. Garfield, as a dynamic in hiring decisions and in awarding grants and tenure. “It never occurred to me that ‘impact’ would become so controversial,” says Dr. Garfield.

Over the past 10 years, IF has come to routinely be applied in ways that Dr. Garfield never imagined. “The term ‘impact factor’ has gradually evolved, especially in Europe, to describe both journal and author impact,” he says.

click for large version

Repercussions of Rank

According to Michael D. Lockshin, MD, editor of A&R, in countries such as England, Germany, France, and China, laboratory space, promotions, and budgets can be based on the IF of the journals in which an author publishes “Academics in these countries are desperate to be in high–impact factor journals,” says Dr. Lockshin, who is professor of medicine and obstetrics-gynecology at Weill Medical College and an attending physician at the Hospital for Special Surgery in New York City. In Spain, researchers have a legal right to be rewarded for publishing in high-impact journals, and in China some universities require PhD candidates to publish two articles in high-impact journals before they are allowed to earn their doctorate.

Although this kind of application is not as prevalent in the United States, other issues are influenced by IF. Dr. Garfield has wryly referred to “Citation Sanity and Insanity: The Obsession and Paranoia of Citations and Impact Factor” as an alternative title to an address he gave in 2005 to the International Congress of Peer Review and Biomedical Publication.¹

The allegations against this system run from claims that IF reporting can be artificially manipulated by editors who are trying to raise their numbers to complaints that while academics vie for limited space in high-IF journals, their work is being delayed from reaching the community in the most timely fashion.

European critics say there is an English-language bias to the reporting. With American journals overwhelmingly citing research reported in English, the mean journal impact of American science is 30% above the world average.² This puts pressure on authors around the world to eschew low-IF national journals in favor of a chance to appear in high-IF English-language publications. Papers that do appear in low-IF national journals are often underrated by the scientific community and policy makers.

The ability of pharmaceutical companies and other potential advertisers to employ IF in their buying decisions ups the ante. Anecdotal reports of self-citations have editors, eager to boost their reputation, citing their own publication in editorials and suggesting that authors add citations from their journal to upcoming articles. Fortunately, according to Dr. Lockshin, Thomson Scientific sees these efforts as transparent and seeks to discount self-citations.

Controversial Calculation

Another issue is the mix of articles in a publication and manner in which Thomson Scientific arrives at the final calculation. Clinical studies, clinical definitions/consensus reports, and review articles are highly desirable because they generate the most citations. A single sensational article can increase IF more than several well-regarded basic science papers, so editors can be tempted to alter their publication mix to raise their IF. Consideration as to which authors and articles will contribute the most to IF could affect the information that ultimately gets to the readership.

Letters to the editor, news articles, editorials, book reviews, and abstracts of meetings are not counted as citable articles, but can raise the IF by increasing the equation’s numerator. In June 2006, the Public Library of Science (PLoS) issued this editorial opinion: “We conclude that science is currently rated by a process that is itself unscientific, subjective, and secretive.”³

Although, according to Dr. Lockshin, the definition of “citable” is arbitrary, because journals are not uniform in article structure, he takes exception to the PLoS view. “Allowing for the many variations of journal and citation style, we judge that [the Thomson process] is an honest and fair attempt, and as sound as any we could devise, to rate journals by an open and explicable metric,” he says.

Perhaps the most resonant argument is that IF does not take into account the quality of the work in individual articles or the utility of a journal to readers. Citations may come from controversy, retractions, and authors who refute the findings of an article. Antony Rosen, MD, director of the rheumatology division at Johns Hopkins University in Baltimore and chair of the ACR Journal Publications Committee, views IF in a pragmatic light. “In terms of relevance, quality, and importance, there are lots of useful articles that never get cited,” Dr. Rosen says. “Impact factor is just one measure of a journal, nothing more.”

Francine Kaplan is a medical journalist based in Atlanta.

References

1. Garfield E. The Agony and the Ecstasy: The history and meaning of the journal impact factor. International Congress on Peer Review and Biomedical Publication. Chicago, September 16, 2005.
2. Braun T, Glänzel W, Grupp H. The scientometric weight of 50 nations in 27 science areas, 1989–1993. Part II. Life sciences. Scientometrics. 1996;34:207-237.
3. The Impact Factor Game. It is time to find a better way to assess the scientific literature. PLoS Med. 2006;3(6):e291.