I’ve certainly written about some pretty sketchy scientific studies during my career, although I hope I’ve gotten sharper-eyed in recent years at spotting — and then ignoring — the more questionable ones.
Yes, yes, a nutritious diet — particularly one rich in fruits and vegetables — will contribute to your overall health. But no single food is going to sway your health one way or the other.
So, I didn’t fall into the trap (fortunately) set by science reporter John Bohannon who, at the request of a German TV reporter working on a documentary about the junk-science diet industry, recently conducted and got published (in a so-called medical journal) a real but actually bogus study about chocolate and weight loss.
He wrote about the entire dismal experience, including how he got the media to publicize his study’s ridiculous results, on the io9.com website earlier this week.
“My colleagues and I recruited actual human subjects in Germany,” he writes. “We ran an actual clinical trial, with subjects randomly assigned to different diet regimes. And the statistically significant benefits of chocolate that we reported are based on the actual data. It was, in fact, a fairly typical study for the field of diet research. Which is to say: It was terrible science. The results are meaningless, and the health claims that the media blasted out to millions of people around the world are utterly unfounded.”
Bohannon isn’t new to this variation of gonzo journalism, as he himself explains:
[Two years ago] I had run a sting operation for Science on fee-charging open access journals, a fast-growing and lucrative new sector of the academic publishing business. To find out how many of those publishers are keeping their promise of doing rigorous peer review, I submitted ridiculously flawed papers and counted how many rejected them. (Answer: fewer than half.)
Still, even he was surprised by how easily he was able to fool so many people with his latest fake-research caper.
A ‘dirty little science secret’
Bohannon describes in his article exactly why his study — which followed 15 people for 21 days after randomly assigning them to one of three diet groups (low-carb, low-carb plus a 1.5-ounce bar of chocolate daily, or a control group that kept to their current diet) — was meaningless. One of those reasons, he says, is this “dirty little science secret”:
If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements — weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc. — from 15 people. … That study design is a recipe for false positives.
Think of the measurements as lottery tickets. Each one has a small chance of paying off in the form of a “significant” result that we can spin a story around and sell to the media. The more tickets you buy, the more likely you are to win. We didn’t know exactly what would pan out — the headline could have been that chocolate improves sleep or lowers blood pressure — but we knew our chances of getting at least one “statistically significant” result were pretty good. …
You might as well read tea leaves as try to interpret our results. Chocolate may be a weight loss accelerator, or it could be the opposite. You can’t even trust the weight loss that our non-chocolate low-carb group experienced versus control. Who knows what the handful of people in the control group were eating? We didn’t even ask them.
Those huge design flaws in the study didn’t stop a medical journal from publishing it, however. Here’s how that happened:
We needed to get our study published pronto, but since it was such bad science, we needed to skip peer review altogether. Conveniently, there are lists of fake journal publishers. (This is my list, and here’s another.) Since time was tight, I simultaneously submitted our paper — “Chocolate with high cocoa content as a weight-loss accelerator” — to 20 journals. Then we crossed our fingers and waited.
Our paper was accepted for publication by multiple journals within 24 hours. Needless to say, we faced no peer review at all. The eager suitor we ultimately chose was the the International Archives of Medicine. It used to be run by the giant publisher BioMedCentral, but recently changed hands. The new publisher’s CEO, Carlos Vasquez, emailed Johannes [the first name Bohannon used as co-author of the study] to let him know that we had produced an “outstanding manuscript,” and that for just 600 Euros it “could be accepted directly in our premier journal.”
The next step, says Bohannon, was to write a press release with “a sexy lede, a clear nut graf, some punchy quotes, and a kicker.” His documentary partners also put together a video to go with the release — and even hired some people through the Internet to write an acoustic ballad and a rap song to accompany it.
When reporters contacted me at all, they asked perfunctory questions. “Why do you think chocolate accelerates weight loss? Do you have any advice for our readers?” Almost no one asked how many subjects we tested, and no one reported that number. Not a single reporter seems to have contacted an outside researcher. None are quoted.
That’s a lot of media hype. Yet, the only major U.S. media outlet that picked up the story seems to have been Slate magazine and (perhaps) Men’s Health.
I’ll take that as a good sign.
Although, as Gary Schwitzer, publisher of the Minnesota-based HealthNewsReview.org, pointed out on Thursday, Bohannon’s latest escapade is yet “another example of how easy it is to get slipshod studies published, and now, how easy it is to get naive news coverage of a slipshod study.”
People who are desperate for reliable information face a bewildering array of diet guidance — salt is bad, salt is good, protein is good, protein is bad, fat is bad, fat is good — that changes like the weather. But science will figure it out, right? Now that we’re calling obesity an epidemic, funding will flow to the best scientists and all of this noise will die down, leaving us with clear answers to the causes and treatments.
Or maybe not. Even the well-funded, serious research into weight-loss science is confusing and inconclusive…
The public relies on journalists to help them get through all that research “noise,” says Bohannon, but too often they (we) fail.
“The … problem with the diet science beat is that it’s science,” he writes. “You have to know how to read a scientific paper — and actually bother to do it. For far too long, the people who cover this beat have treated it like gossip, echoing whatever they find in press releases. Hopefully our little experiment will make reporters and readers alike more skeptical.”
Update: Late on Thursday, the editor of International Archives of Medicine posted the following statement on the journal’s Facebook page:
Disclaimer: Weeks ago a manuscript that was being reviewed in the journal "Chocolate with High Cocoa Content as a Weight-Loss Accelerator" appeared as published by mistake. Indeed that manuscript was finally rejected, although it went online for some hours.
We are sorry for the inconvenience. We are taking measures to avoid this kind of mistakes happens again.