I’ve certainly written about some pretty sketchy scientific studies during my career, although I hope I’ve gotten sharper-eyed in recent years at spotting — and then ignoring — the more questionable ones.
High on the list of the types of studies I now avoid are those that claim a specific food “fights cancer” or “helps shed pounds” or “boosts memory.”
Yes, yes, a nutritious diet — particularly one rich in fruits and vegetables — will contribute to your overall health. But no single food is going to sway your health one way or the other.
So, I didn’t fall into the trap (fortunately) set by science reporter John Bohannon who, at the request of a German TV reporter working on a documentary about the junk-science diet industry, recently conducted and got published (in a so-called medical journal) a real but actually bogus study about chocolate and weight loss.
He wrote about the entire dismal experience, including how he got the media to publicize his study’s ridiculous results, on the io9.com website earlier this week.
“My colleagues and I recruited actual human subjects in Germany,” he writes. “We ran an actual clinical trial, with subjects randomly assigned to different diet regimes. And the statistically significant benefits of chocolate that we reported are based on the actual data. It was, in fact, a fairly typical study for the field of diet research. Which is to say: It was terrible science. The results are meaningless, and the health claims that the media blasted out to millions of people around the world are utterly unfounded.”
Bohannon isn’t new to this variation of gonzo journalism, as he himself explains:
[Two years ago] I had run a sting operation for Science on fee-charging open access journals, a fast-growing and lucrative new sector of the academic publishing business. To find out how many of those publishers are keeping their promise of doing rigorous peer review, I submitted ridiculously flawed papers and counted how many rejected them. (Answer: fewer than half.)
Still, even he was surprised by how easily he was able to fool so many people with his latest fake-research caper.
A ‘dirty little science secret’
Bohannon describes in his article exactly why his study — which followed 15 people for 21 days after randomly assigning them to one of three diet groups (low-carb, low-carb plus a 1.5-ounce bar of chocolate daily, or a control group that kept to their current diet) — was meaningless. One of those reasons, he says, is this “dirty little science secret”:
If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements — weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc. — from 15 people. … That study design is a recipe for false positives.
Think of the measurements as lottery tickets. Each one has a small chance of paying off in the form of a “significant” result that we can spin a story around and sell to the media. The more tickets you buy, the more likely you are to win. We didn’t know exactly what would pan out — the headline could have been that chocolate improves sleep or lowers blood pressure — but we knew our chances of getting at least one “statistically significant” result were pretty good. …
You might as well read tea leaves as try to interpret our results. Chocolate may be a weight loss accelerator, or it could be the opposite. You can’t even trust the weight loss that our non-chocolate low-carb group experienced versus control. Who knows what the handful of people in the control group were eating? We didn’t even ask them.
Quick acceptance
Those huge design flaws in the study didn’t stop a medical journal from publishing it, however. Here’s how that happened:
We needed to get our study published pronto, but since it was such bad science, we needed to skip peer review altogether. Conveniently, there are lists of fake journal publishers. (This is my list, and here’s another.) Since time was tight, I simultaneously submitted our paper — “Chocolate with high cocoa content as a weight-loss accelerator” — to 20 journals. Then we crossed our fingers and waited.
Our paper was accepted for publication by multiple journals within 24 hours. Needless to say, we faced no peer review at all. The eager suitor we ultimately chose was the the International Archives of Medicine. It used to be run by the giant publisher BioMedCentral, but recently changed hands. The new publisher’s CEO, Carlos Vasquez, emailed Johannes [the first name Bohannon used as co-author of the study] to let him know that we had produced an “outstanding manuscript,” and that for just 600 Euros it “could be accepted directly in our premier journal.”
Although the Archives’ editor claims that “all articles submitted to the journal are reviewed in a rigorous way,” our paper was published less than 2 weeks after Onneken’s credit card was charged. Not a single word was changed.
Reeling in the media
The next step, says Bohannon, was to write a press release with “a sexy lede, a clear nut graf, some punchy quotes, and a kicker.” His documentary partners also put together a video to go with the release — and even hired some people through the Internet to write an acoustic ballad and a rap song to accompany it.
It worked:
We landed big fish before we even knew they were biting. Bild rushed their story out — ”Those who eat chocolate stay slim!” — without contacting me at all. Soon we were in the Daily Star, the Irish Examiner, Cosmopolitan’s German website, the Times of India, both the German and Indian site of the Huffington Post, and even television news in Texas and an Australian morning talk show.
When reporters contacted me at all, they asked perfunctory questions. “Why do you think chocolate accelerates weight loss? Do you have any advice for our readers?” Almost no one asked how many subjects we tested, and no one reported that number. Not a single reporter seems to have contacted an outside researcher. None are quoted.
That’s a lot of media hype. Yet, the only major U.S. media outlet that picked up the story seems to have been Slate magazine and (perhaps) Men’s Health.
I’ll take that as a good sign.
Although, as Gary Schwitzer, publisher of the Minnesota-based HealthNewsReview.org, pointed out on Thursday, Bohannon’s latest escapade is yet “another example of how easy it is to get slipshod studies published, and now, how easy it is to get naive news coverage of a slipshod study.”
“When you critique (and try to improve) health news every day, as we have on HealthNewsReview.org for more than 9 years, you find broad, deep problems with the way science is communicated to the public,” Schwitzer added in a guest post for Retraction Watch.
Adding to the confusion
Why does this all matter? Writes Bohannon:
People who are desperate for reliable information face a bewildering array of diet guidance — salt is bad, salt is good, protein is good, protein is bad, fat is bad, fat is good — that changes like the weather. But science will figure it out, right? Now that we’re calling obesity an epidemic, funding will flow to the best scientists and all of this noise will die down, leaving us with clear answers to the causes and treatments.
Or maybe not. Even the well-funded, serious research into weight-loss science is confusing and inconclusive…
The public relies on journalists to help them get through all that research “noise,” says Bohannon, but too often they (we) fail.
“The … problem with the diet science beat is that it’s science,” he writes. “You have to know how to read a scientific paper — and actually bother to do it. For far too long, the people who cover this beat have treated it like gossip, echoing whatever they find in press releases. Hopefully our little experiment will make reporters and readers alike more skeptical.”
Hopefully.
You can read Bohannon’s article on the io9.com website.
Update: Late on Thursday, the editor of International Archives of Medicine posted the following statement on the journal’s Facebook page:
Disclaimer: Weeks ago a manuscript that was being reviewed in the journal “Chocolate with High Cocoa Content as a Weight-Loss Accelerator” appeared as published by mistake. Indeed that manuscript was finally rejected, although it went online for some hours.
We are sorry for the inconvenience. We are taking measures to avoid this kind of mistakes happens again.
rip, read and write
One used to be able to ascribe these things to simple journalistic laziness. In the era of video news releases, radio actualities, and sponsored magazine inserts, wraparound sections and mid-morning shows — not to mention astroturfing, sockpuppetry and flashmobbery — the motives are more sinister.
“Both sides do it”
There is also an obsessive concern with an odd type of “fairness.” This type of fairness requires that claims or assertions must be countered by a quote from the opposing side. The journalist’s job has gone from reporting the truth to reporting what is being said.
As Paul Krugman has said, an article that points out that the earth is round would have to include a quote from a flat-earther, and the headline would be “Opinions Vary on the Shape of the Earth.”
The best part
…of this spoof was the acoustic ballad, “The Chocolate Transformation.”
More spoof studies should come with them.
Who is misleading the public?
The headline of your story says that bad science is misleading the public. But between the bad studies published without peer review in pay-to-publish journals and the public are bad science reporters and under- and poorly-staffed media outlets. The Strib recently published, without comment, an article from the NYTimes that the Times public editor had called out for poor methodology. Poorly written, sensationalistic reports of questionable studies get passed around from outlet to outlet without vetting. I submit that the media are responsible for spreading junk science around, not the study authors.