Confessions extracted under torture are notoriously unreliable. A new study in the New England Journal of Medicine illustrates this point well.
The study analyzes the relationship between weight and mortality risk. In particular, it tries to determine whether being “overweight” (this is currently defined by our public health authorities as 146 to 174 pounds for an average height woman, and 174 to 208 pounds for an average height man) is associated with an increased risk of death.
This is an especially controversial issue for two reasons. First, most Americans who the government claims weigh too much are in this “overweight” category. Second, many studies find either that there is no increased mortality risk associated with being “overweight,” or indeed that the risk of death among the so-called “overweight” is actually lower than among so-called “normal weight” individuals.
In particular, a 2005 study led by CDC researcher Katherine Flegal found 86,000 excess deaths per year in the United States among “normal weight” people, when comparing their mortality risk to that of so-called “overweight” persons. Because of the current panic over fat, this study caused quite a furor, even though its findings were consistent with many other investigations of the same issue.
It seems the authors of the new study in the New England Journal of Medicine were determined to refute Flegal’s findings _ even if they had to subject their data to techniques that violate the scientific equivalent of the Geneva Convention.
The researchers collected data from 527,265 AARP members, who were followed for ten years. What they found was exactly the same result reported by Flegal and her colleagues: Among both men and women, “overweight” people had the lowest mortality risk. This result, however, was clearly unacceptable. So they began torturing their data.
First, they threw out any subjects who had ever smoked. The justification for doing so in studies exploring the relationship between weight and health is that some people smoke to remain thin, so increased health risk among thin people may be a product of smoking rather than thinness. (In fact, in this study the percentage of “normal weight” people who had never smoked was higher than the percentage of “overweight” and “obese” never-smokers, but never mind.)
Yet even after limiting their analysis to never-smokers, the authors found no increased mortality risk among the “overweight” when compared to so-called “normal weight” people. So they then engaged in the methodological equivalent of waterboarding. Bizarrely, rather than using the weights of the subjects at the time they entered the study, the authors asked the subjects what they had weighed at age 50, and used this weight instead (all the subjects were over 50 at the study’s start, and some were as old as 71; 40 percent did not even respond to the question about what they weighed at age 50, which says something about the reliability of the responses the authors did get).
This, at last, produced a (modest) increase in mortality risk associated with “overweight,” thus allowing the authors to draw their conclusion that “overweight is associated with an increased risk of death.”
But notice how this result was produced. Since the “overweight” people in the study still had the lowest death risk _ even after the authors tossed out 70 percent of their subject pool by limiting their analysis to never-smokers _ the study found “overweight” associated with an increased risk of death only among a particular subgroup: people who had been “overweight” at age 50, but were at a “normal weight” when they later entered the study.
In other words, what the study really found is that for middle-aged “overweight” people weight loss increases the risk of death significantly! (This, by the way, is a very common finding in studies of this sort.)
The authors, needless to say, fail to note this awkward fact, which does not merely contradict, but actually inverts, the public health message their study is intended to bolster. Will journalists covering the study manage to figure this out on their own? Fat chance.
(Paul Campos is a law professor at the University of Colorado and can be reached at Paul.Campos(at)Colorado.edu.)