Scientometrics & the Study of Being Wrong
“Half of what you will learn is wrong. You just don’t know which half.”
I assume someone recited this to me during medical school. If I’m being honest, however, I’m really not sure. For me, it is probably part of the revisionist narrative we all create for ourselves, replete with half-truths and stories of dubious provenance.
It turns out this phrase may be just such an example of revisionist history. Not knowing which half of your efforts were wasted, I thought, was specific to medical training. Apparently, this observation originally belonged to the field of advertising.
John Wanamaker was an American entrepreneur who founded the first department store in Philadelphia—if his name sounds vaguely familiar to you, it may be because of the fingerprints he left on that great city, in the form of the Wanamaker building and the Wanamaker Grand Court Organ. Among other contributions, he pioneered the use of advertising in marketing. Before Wanamaker, department stores merely offered goods and services; afterward, department stores used advertising to tell you what you wanted to buy.
Wanamaker knew better than most that you couldn’t predict the effect of advertising dollars. In fact, he said, “Half of the money I spend on advertising is wasted; the trouble is I don’t know which half.”4
Sound familiar?
This has given birth to an entire field of study. Scientometrics, or the measurement of science, is based on the philosophy that the truth has a half-life. As time marches on, certain truths become obsolete. Although this seems counterintuitive, medicine pioneered this concept: The entire idea of continuing medical education acknowledges that facts are neither static nor absolute.
In 1962, the economist Fritz Machlup coined the phrase the half-life of knowledge to denote the time it takes for half of knowledge in a given field to be superseded by a new set of truths.5 In the 1920s, the half-life of an engineering degree was 35 years. In the 1960s, the half-life of the same degree was about 10 years. Now, it’s about five years.6
The entire idea of continuing medical education acknowledges that facts are neither static nor absolute.
Surgery is not immune to this phenomenon. In 1997, John C. Hall and Cameron Platell selected abstracts from the journal Surgical Gynecology and Obstetrics, from 1935 to 1994. Based on their review, they estimated that the rate of the loss of truth is 0.75% per year; based on this, they estimated the half-life for the truth of statements in the surgical literature is 45 years.7
It seems that medicine is doomed to the same fate. In 2002, Poynard et al. examined articles on cirrhosis or hepatitis published between 1945 and 1999. As of 2000, 60% of the conclusions were still presumed to be true, 19% were obsolete, and 21% were demonstrably wrong. Interestingly, this analysis led to the same estimate for the half-life of the truth: 45 years.8
Part of the problem is the rate at which knowledge is expanding. R. Buckminster Fuller estimated that until 1900, human knowledge doubled approximately once every century. By 1925, he asserted that knowledge doubled every 25 years, and by 1982, it doubled once every year. How this acceleration may affect the half-life of medical truth is anyone’s guess.9
Also, we are bad at predicting which truths will survive the test of time. The Poynard et al. study indicated that high-quality studies were no more likely to yield long-lived truths than low-quality studies. One interesting finding was that the 50-year survival of negative studies was significantly higher than that of positive studies. In other words, we are better at knowing when we are wrong than when we are right.
It is a sign of progress that this problem is now affecting rheumatology, as well. The world of vasculitis is going through conniptions trying to come to terms with the results of the PEXIVAS study, which demonstrated that plasma exchange did not benefit patients with severe forms of ANCA-associated vasculitis.10 Some of us are going through the predictable stages of denial: Perhaps the patients enrolled in the study weren’t sick enough. Perhaps we didn’t get them on plasma exchange fast enough. Perhaps it only makes a difference if we didn’t pulse patients with steroids. Perhaps it only works for patients who were dialysis dependent or oxygen dependent or had a strong craving for Italian food.
It’s hard to know whether there may be a kernel of truth to any of these misgivings. For many of us, plasma exchange for patients with severe vasculitis has been an article of faith, and it is difficult for faith to give way to mere fact. Given the half-life of the truth, it is possible that 45 years from now, our understanding of plasma exchange will be replaced by another truth. It seems equally possible, however, that my fellow will be citing PEXIVAS to her trainee only to be greeted by peals of laughter, echoing down the hospital halls.
