UCare generously supports MinnPost’s Second Opinion coverage; learn why.

Why we should be highly skeptical of medical research

I didn’t want November to end without pointing Second Opinion readers to a terrific article published in this month’s issue of The Atlantic magazine. Reporter David H. Freedman profiles Greek physician and “meta-researcher” John Ioannidis, who for the last decade or so has been voraciously challenging the credibility of medical research.

After reading the article, you’ll have a much more skeptical — and thus healthier — attitude toward the much-publicized findings of medical-research studies.

Writes Freedman:

[Ioannidis] and his team have shown, again and again, and in many different ways, that much of what biomedical researchers conclude in published studies — conclusions that doctors keep in mind when they prescribe antibiotics or blood-pressure medication, or when they advise us to consume more fiber or less meat, or when they recommend surgery for heart disease or back pain — is misleading, exaggerated, and often flat-out wrong. He charges that as much as 90 percent of the published medical information that doctors rely on is flawed. His work has been widely accepted by the medical community; it has been published in the field’s top journals, where it is heavily cited; and he is a big draw at conferences.
Given this exposure, and the fact that his work broadly targets everyone else’s work in medicine, as well as everything that physicians do and all the health advice we get, Ionnidis may be one of the most influential scientists alive. Yet for all his influence, he worries that the field of medical research is so pervasively flawed, and so riddled with conflicts of interest, that it might be chronically resistant to change — or even to publicly admitting that there’s a problem.

Even randomized trials, considered the most reliable type of study, end up being wrong up to 25 percent of the time, Ioannidis has found. Unfortunately, as Ioannidis reported in a 2005 paper, the medical community too often fails to notice research that disproves earlier findings (or vehemently rejects it — consider the anger unleashed last year by doctors as well as patients over revised mammography screening recommendations). Writes Freedman:

[Ioannidis] zoomed in on 49 of the most highly regarded research findings in medicine over the previous 13 years, as judged by the science community’s two standard measures: the papers had appeared in the journals most widely cited in research articles, and the 49 articles themselves were the most widely cited articles in these journals. These were articles that helped lead to the widespread popularity of treatments such as the use of hormone-replacement therapy for menopausal women, vitamin E to reduce the risk of heart disease, coronary stents to ward off heart attacks, and daily low-dose aspirin to control blood pressure and prevent heart attacks and strokes.
Ionnidis was putting his contentions to the test not against run-of-the-mill research, or even merely well-accepted research, but against the absolute tip of the research pyramid. Of the 49 articles, 45 claimed to have uncovered effective interventions. Thirty-four of these claims had been rested, and 14 of these, or 41 percent, had been convincingly shown to be wrong or significantly exaggerated. If between a third and a half of the most acclaimed research in medicine was proving untrustworthy the scope and the impact of the problem were undeniable.

One of the big, underlying problems of medical research is bias. “At every step in the process, there is room to distort results, a way to make a stronger claim or to select what is going to be concluded,” Ionnidis told Freedman. “There is an intellectual conflict of interest that pressures researchers to find whatever it is that is most likely to get them funded.” (And with much of that research now funded by drug companies, the pressure to get the “right” results is even greater.)

You can read Freedman’s entire article here.

Comments (6)

  1. Submitted by Arvonne Fraser on 11/29/2010 - 11:52 am.

    Thanks, Susan Perry, for an excellent article. Americans seem to jump at the next new thing, doctors included, when we should be jumping at trying to save health care dollars. Every new thing has costs to all of us, making thousands run to a doctor or clinic for the latest. It then becomes something to talk about over lunch or wherever, and more run seeking nirvana.

    If we hope to control health care costs, we better–each one of us–accept the fact that good health habits, including exercise and keeping busy mentally, are fare cheaper than running to the doctor for the latest thing.

  2. Submitted by Bill Gleason on 11/29/2010 - 01:52 pm.

    Ms. Fraser’s point is well taken: that the public needs to take some control over medical costs. A glaring example is the enormous waste of money that will result from the Children’s Hospital competition that is ongoing in the Twin Cities. At one place the cost per bed is $1 million. Of course this is rationalized by claiming that this is a research hospital. Meanwhile the other Children’s Hospital is laying off 250 people.

    But I am curious, Susan, about your take home message for the situation described above. I certainly share your skepticism, but what is the average person to make of this situation? It appears from what you have said that around half of the new developments are not backed up by reliable research.

    What do you think should be done about this situation? I realize that your response might be complex, perhaps the subject of another excellent article?

    Bill Gleason, U of M med school faculty and alum

  3. Submitted by Susan Perry on 11/29/2010 - 05:34 pm.

    Bill:

    I am highly skeptical of every medical study I read. I’m not going to stop reporting on such studies, as their publication is news. But I try to point out the limitations of the research–and conflicts of interest. Financial conflicts of interest have become a huge, huge problem in medical research–and in medical practice.

    I believe that in a hundred years, people will look upon this time in medicine as we look upon the medicine of the 19th century–woefully inadequate and all too often driven by wishful thinking (and the profit motive) rather than by scientific fact.

    It can take the medical community decades to let go of a long-held but erroneous belief.

  4. Submitted by Hénock Gugsa on 11/29/2010 - 10:12 pm.

    Thanks, Susan, for this article.

    It is perfectly-timed as we will soon be bombarded by the publications or announcements of findings by research groups and foundations of one field or another. Yes, indeed, between now and almost February, we can expect revelations of new miracle cures, or reinstatement of debunked old ones.

    Today, I read in the WSJ (*) that bio-medical researchers have made significant progress in their work on an anti-aging enzyme (a kind of a “fountain of youth”.) Shangri-La here already, I doubt it!

    I’m sure we all agree that moderate skepticism is always a good (healthy) thing in every arena of human endeavor. Was it Alexander Pope who said, “Blessed is he who expects nothing, for he shall never be disappointed.”

    ____________________________________________

    (*)Read article titled, “Aging Ills Reversed in Mice” in the Wall Street Journal –> http://online.wsj.com/article/SB10001424052748703785704575642964209242180.html?mod=WSJ_hp_MIDDLENexttoWhatsNewsSecond

  5. Submitted by Greg Kapphahn on 11/30/2010 - 07:47 am.

    Perhaps a useful course of action would be to ignore any medication you see advertised heavily. Time and time again these newest, most expensive drugs have been pulled from the market after important, even deadly long term side effects were “discovered” (which it always turns out were noticed in the research studies but minimized or ignored in their final approval).

    Another useful approach involves considering how your physician operates. If he/she is willing to prescribe you whatever you want, you might be wiser to find one a bit more skeptical of every new miracle drug, medical procedure, or medical test that comes down the pipe. If you can’t imagine that your doctor would ever say to you that the meds you’re on are just as effective and far less expensive than the one you just saw on TV, find yourself another doctor.

    The medical products companies and pharmaceutical companies are trying to make as much money as possible. Through their advertising, they’re trying to turn us into a nation of hypochondriacs and using the least skeptical, most cooperative doctors to help them do so. Find yourself a doctor who’s a bit too skeptical to be a too-willing partner in that enterprise.

  6. Submitted by Paul Udstrand on 11/30/2010 - 10:34 am.

    Actually there are two problems here that Ms. Perry fails to account for.

    One is ignorance regarding the scientific process itself. Medical research is part of an ongoing scientific process no different than any other scientific process. Approaches to medical knowledge that treat this process as if it’s some kind of back and forth debate between various researchers completely distort reality. Scientific knowledge is a product of consensus, all of these various published studies are part of that process. Mr. Loannidis’s observations are rather mundane. All scientists know that research is problematic and all studies contain flaws. That’s why studies are replicated and duplicated. THAT’s why you have soooooo many studies being conducted. Medical knowledge is not determined by any given study, it’s the over-all trends produced by multiple studies over time that produce stable and reliable observations. It’s the constant re-examining of observations that reveal problems with previous observations.

    The difficult nature of some research, and the possible unreliability of some observations are the very reason that results contrary to predominate assumptions are challenged. It is a basic tenet of science that the burden of proof lies with those who are challenging the existing paradigm, the more extraordinary the challenge, the more extraordinary the evidence must be. This is not evidence of closed mindedness, it ensures scientific rigor. New or contrary results are not so much “attacked” as they are challenged, the challenge is make sure the data are reliable. Without these challenges Mr. Loannididis would have no data to analyze one way or the other.

    By the way, one should always remember that meta-statistics is a fickle mistress, play with her at your peril. The big data-mining revolution everyone was expecting a few years ago has failed to materialize. Turns out the world is a lot more complicated than one would gather from watching “Numbers” on television. Loannidids may be a popular speaker, but that does not make him a vessel of knowledge and wisdom, especially if he’s not doing any original work of his own. We all stand on the shoulders of giants, but meta-statisticians tend to lose their balance more often than most. Ask me where Loannididis is three years from now.

    Beyond the scientific ignorance there is the problem of the media. You have this ongoing process of scientific study that produces published articles. The media, and science or health “writers” constantly focus on trees at the expense of the forest. Judging from media accounts one could not be forgiven for thinking that doctors are changing their minds about diet and exercise every other week. This is because individual studies are reported completely devoid of context within the over-all scientific discourse. We constantly hear or read about new “studies” as if they are headlines from the scientific community. The reality is that any given study is one of many within a given field of inquiry that may or may not successfully challenge existing assumptions.

    As for medical practice, I’m sorry but anyone who thinks practicing doctors are as confused about medical research as the public is needlessly worried. Yes, doctors stay apprised of the ongoing research, but they understand the nature of the process and stick with the consensus for the most part. That’s a good thing. You don’t want your doctor changing his treatments every other day based on a single study that hasn’t been replicated or properly vetted. That’s a prescription for medical chaos. Everyone knows their knowledge is incomplete and possibly flawed but you have to work with what you have.

    In medicine and science there’s a very basic principle… beware that you don’t know what you don’t know. The lack of stability in medical science simply reflects the complexity of the universe. No one knows everything. The fact that medical knowledge is constantly being revised i.e. 25% of research turns out to be wrong… is a good thing. If your expecting the day to come when doctors will say: “well that’s it, we know everything there is to know and we’re absolutely certain we got it right” you going to have to pass that wait on to your ancestors because it’s not on the calender. The constant revision of knowledge is a good thing, it’s why you can expect to live twice as long today as you would have 40 years ago.

    Should you be skeptical of scientific research? Duh. Skepticism is the very core scientific inquiry. Skepticism is why we’re still doing science. That’s the difference between science and religion. Just remember skepticism isn’t denial or distrust, it’s the reservation of judgment until sufficient information allows you to render a conclusion you think you can rely on.

Leave a Reply