As Belluz explains, outcome switching has long been used to dupe both doctors and the public about the effectiveness of various prescription drugs. One of the most infamous cases involved a clinical trial known as Study 329. That study, funded by the drug company GlaxoSmithKline (GSK) and published in 2001, claimed to show that the antidepressant paroxetine, or Paxil, was “well tolerated and effective” for kids.
But years later — after doctors had written 2 million Paxil prescriptions for children and adolescents — “it became clear that the study’s original conclusions were wildly wrong,” writes Belluz. “Not only is Paxil ineffective, working no better than placebo, but it can actually have serious side effects, including self-injury and suicide.”
Belluz explains how the study’s authors managed to fool everybody:
Before researchers start clinical trials, they’re supposed to pre-specify which health outcomes they’re most interested in. For an antidepressant, these might include people’s self-reports on their mood, how the drug affects sleep, sexual desire, and even suicidal thoughts.
The idea is that researchers won’t just publish positive or more favorable outcomes that turn up during the study, while ignoring or hiding important results that don’t quite turn out as they were hoping.
But that doesn’t always happen. “In Study 329,” explains Ben Goldacre, a crusading British physician and author, “none of the pre-specified analyses yielded a positive result for GSK’s drug, but a few of the additional outcomes that were measured did, and those were reported in the academic paper on the trial, while the pre-specified outcomes were dropped.”
And, yes, it’s certainly OK to report on unexpected outcomes in studies. That’s one of the ways science moves forward. But it’s not OK to simply drop any mention of the drug’s effects on the pre-specified outcomes just because the results weren’t positive.
“Switching your outcomes breaks the assumptions in your statistical tests,” Goldacre explains to Belluz. “It allows the ‘noise’ or ‘random error’ in your data to exaggerate your results (or even yield an outright false positive, showing a treatment to be superior when in reality it’s not).”
“We do trials specifically to detect very modest differences between one treatment and another,” he adds. “You don’t need to do a randomized trial on whether a parachute will save your life when you jump out of an airplane, because the difference in survival is so dramatic. But you do need a trial to spot the tiny difference between one medical intervention and another. When we get the wrong answer, in medicine, that’s not a matter of academic sophistry— it causes avoidable suffering, bereavement, and death. So it’s worth being as close to perfect as we can possibly be.”
Holding researchers accountable
Goldacre, with the help of a team of medical students at the University of Oxford, launched a new initiative in October called the Compare Project. They are “systematically checking every trial published in the top five medical journals, to see if they have misreported their findings.” (Those five journals are the New England Journal of Medicine, the Journal of the American Medical Association, The Lancet, the Annals of Internal Medicine and BMJ.)
As of yesterday, Jan. 4, the project had checked 66 clinical trials. Nine of the trials were “perfect” — in other words, the studies reported all of their pre-specified outcomes and didn’t add any new ones. But in the other trials, 355 pre-specified outcomes were not reported and 336 new outcomes were silently added.
“On average, each trial reported just 58.2% of its specified outcomes,” Goldacre and his team write on the Compare Project website. “And on average, each trial silently added 5.1 new outcomes.”
When Goldacre and his team uncover evidence of outcome switching, they write a letter about it to the academic journal — and then track if the letter gets published. Of the 55 letters sent thus far, five have been published, eight have been rejected for publication, and 24 remain unpublished after four weeks.
As Goldacre makes clear to Belluz, he doesn’t think “every trialist and journal that we’ve caught switching its outcomes is doing so deliberately, to rig the results. Often it’s clumsiness, or a failure to take the issue sufficiently seriously.”
“But this sloppy reporting gives cover to people who are deliberately cherry-picking their results and rigging their findings,” he adds. “That’s why it’s so important to hold the line and strive to report trials correctly, or at least explain why you’ve switched from your original plans.”