UCare generously supports MinnPost’s Second Opinion coverage; learn why.

Psychotherapy’s effectiveness in treating depression may be overstated, study suggests

Creative Commons/Samantha Evans
Psychologist Steven Hollon: “Psychotherapy does work. It just doesn’t work as well as you would think from reading the scientific literature.”

Talk therapy for the treatment of depression may not be as effective as the published research literature currently suggests, according to the authors of a study that appears this week in the journal PLoS ONE.

A few years ago, a team of researchers — led by one of the authors of the new study — made a similar finding regarding antidepressant drugs.

The reason for the overestimations of the benefits of both treatment approaches is a phenomenon known as publication bias. It’s the tendency of researchers to be more likely to submit — and medical journals to accept — manuscripts for publication when the study has strong, positive outcomes rather than weak or negative ones. 

Publication bias results in published studies presenting a more favorable view of a treatment than is actually the case.

“This doesn’t mean that psychotherapy doesn’t work,” said Steven Hollon, one of the authors of the study and a psychologist at Vanderbilt University, in a released statement. “Psychotherapy does work. It just doesn’t work as well as you would think from reading the scientific literature.”

Depression is one of the most common and disabling illnesses in the United States, affecting about 8 percent of people aged 12 or older during any two-week period, according to the U.S. Centers for Disease and Prevention. About 8 million Americans seek some sort of medical care for depression each year.

Study details

For the study, Hollon and his colleagues reviewed 55 grants awarded by the National Institutes of Health (NIH) to fund clinical trials of psychological treatments for depression between 1972 and 2008. The results of 13 of those studies had not been published.

They then contacted the researchers who had conducted the unpublished studies and asked for their results. They were able to get the data for 11 of the studies.

After analyzing the unpublished and published data together, Hollon and his colleagues concluded that the effectiveness of psychotherapy for the treatment of depression was about 25 percent lower than suggested in the published data alone.

That finding mirrors the 24 percent overestimation of the effectiveness of antidepressants for the treatment of depression found in a similarly conducted study in 2008.

‘Not interesting enough’

The research community has increasingly — and sometimes grudgingly — acknowledged publication bias as a major problem, and calls have been made to require all study data to be publicly available, whether or not it ends up being published in a journal.

The journal PLoS One has been a leader in this effort, fully supporting the global AllTrials campaign, which wants all clinical trials to be registered and all results publicly reported.

In the current study, the most common reason cited by researchers for not publishing their results was their belief that their negative (“no benefit”) finding was not interesting enough to justify making the effort to publish it. And, indeed, neither of the two studies showing no benefit that were submitted to a journal was accepted for publication.

Hollon and his colleagues say that researchers — particularly those whose studies are publicly funded — have “fiduciary and ethical duty to provide a return on the public’s investment by making a good faith effort to publish.”

“We therefore call upon the NIH to expand its Public Access Policy so that the public can have full, not partial, access,” they add.

You can read their study in full on the PLoS ONE website.

Comments (9)

  1. Submitted by Pat Berg on 10/02/2015 - 12:13 pm.

    Oh lovely!

    Meds don’t work and psychotherapy doesn’t work.

    Just what does that leave?

    • Submitted by Curt Carlson on 10/02/2015 - 02:04 pm.

      Did you read the article?

      “This doesn’t mean that psychotherapy doesn’t work,” means that you need to read more than the headlines. It also means science journalists need to be more careful in how they report on the results of studies.

      • Submitted by Pat Berg on 10/02/2015 - 08:34 pm.

        Yes I read the article

        A 25% reduction in effectiveness is pretty darn significant to someone looking for help with something as life-impacting as depression, especially when that reduction in effectiveness is apparently the case for BOTH modes of help that people have been advised to seek out if depression is a problem in their lives.

        • Submitted by Paul Brandon on 10/03/2015 - 01:10 pm.

          First

          neither medical nor talk therapy interventions have ever been shown to be effective in more than a minority of instances. Whether that’s 30% of 20%, the best we can say is that they’re the best we have.
          If we spent a small fraction of the cost of F-35 fighters on research, we could find more effective interventions. However, right now there is no better alternative than a combination of medication and behavior therapy, however limited its effectiveness.
          Saying that the problem is serious doesn’t make an intervention more effective.

  2. Submitted by Donald Larsson on 10/02/2015 - 04:05 pm.

    One–or even two sizes don’t fit all

    Even to a layperson like me, it’s apparent that “clinical depression” has a very wide spectrum that reflects various mixtures of environmental and physical factors. I have no idea how fine-grained these and other studies have been, but it would be interesting to see some breakdown of the stats. Are they really asking the right questions?

    • Submitted by Paul Brandon on 10/03/2015 - 02:38 pm.

      Definition

      Severe (clinical) depression is pretty easy to diagnose (like, the person hasn’t gotten out of bed in a month) and clinical level doses of antidepressants are effective in ameliorating (not curing) the condition. Unfortunately, we don’t yet know much about the biological understrate of depression (or rather, too many people know too much, and what they know is mutually incompatible). So it’s not clear if we are dealing with a single problem that has a single solution.
      The kinds of milder ‘situational’ depression that make of the majority of clinical cases are much more varied, and the diagnosis of them much less reliable. So, with little idea of what we’re treating and what its underlying cause is (or even if it is a single entity), it’s not surprising that treatment is relatively ineffective. Most of the drugs used a broadly active ‘feelgood’ medications, not a treatment for an underlying condition.
      Similarly, the talk therapies that have been proven effective are not cures for depression per se, but instruction in various strategies for dealing with the problems of everyday life.
      It’s not like medicine where you can put the micro organisms which cause an infection under a slide and show that a specific medication kills them (and hopefully not the patient as well ;-).

  3. Submitted by Paul Udstrand on 10/02/2015 - 06:18 pm.

    This is ridiculous

    To begin with, the effectiveness of psychotherapy has been questionable for decades, this is NOT a new question. Several studies over the last few decades have revealed little if any difference between therapy subjects and controls.

    Second, this “study” doesn’t actually look at the effectiveness of psychotherapy at all so it doesn’t really offer any viable conclusions about it’s effectiveness one way or the other. The “publication”, Plos One, is dedicated to demonstrating the existence of publication bias and as such is subject to it’s own publication bias.

    Furthermore the methodology is bizarre. You can’t compare published and unpublished data because the unpublished data isn’t actually “results” in any meaningful way. Unpublished studies have not been peer reviewed, for all you know your unpublished “data” is junk data, and there’s no plausible reason to assume that junk data about psychotherapy is MORE reliable than published data.

    Finally, meta statistics is fickle mistress. You can’t just plug numbers into SPSS or SAS and call it a study. The “Meta” doesn’t eliminate the problem of “garbage in garbage out”.

    Finally finally… I don’t trust any publication that claims to be peer reviewed but doesn’t list the reviewers, or even clearly identify the authors.

    • Submitted by Paul Brandon on 10/03/2015 - 08:25 pm.

      A few contrary points….

      The best experimental studies (not case descriptions) have always shown various talk therapies to be effective in no more than a third of cases.
      Second, Plos One is an online publication venue whose purpose is a lot broader than publication bias (the topic of one specific article).
      Third, unpublished results often ARE peer reviewed. Typically, the journal decides that while the results are methodologically sound, its readers are more interested in positive results (sometimes known as the ‘back drawer effect’.). In this case, the authors actually requested the data that both published and unpublished articles were based on to independently evaluate the size of the effect.
      Finally, reviewers of specific articles are not usually listed in scientific publication, to avoid some of the politics that would result (few people would be willing to do reviews if they knew that angry authors — some of them also reviewers — would be climbing on their necks). As is customary, the authors, their affiliations, and potential conflicts of interest are listed.

      I’ve never been a reviewer, but I have submitted articles for publication. I’ve gotten helpful feedback from reviewers, but they’re listed as ‘Reviewer A, Reviewer B’, etc. The number of people expert in a given field of science is usually quite limited, so one often has a good idea who ‘Reviewer A’ is.
      For the same reason, people will end up reviewing each other’s submissions. Again, this is unavoidable, but anonymity does provide at least some buffering.
      Journals will also sometimes print their whole list of reviewers, but not what specific articles they’ve reviewed.

  4. Submitted by Paul Udstrand on 10/05/2015 - 09:37 am.

    Follow up Mr. Brandon

    Thank you for some additional clarity Mr. Brandon. It IS true that reviewers are “anonymous” in may cases in order to encourage frank discussion, especially in fields that are relatively small.

    Somehow when I followed the links I ended up at a website that described the primary mission of PLos ONE as devoted to unmasking publication bias. However you are correct, it does have a broader mission, and is a respectable online journal. This particular article however makes explicit claims regarding publication bias.

    Another mistake I made was claiming that the authors are not listed, however they are clearly listed at the head of the article. My apologies.

    I still have issues with the data set however, unpublished data MAY have been peer reviewed (it depends) but the whole function of the peer review ostensibly is to help make the study worthy of publication. I realize it can be something of a catch 22 but you can’t assume that every unpublished study is just as solid or rigorously reviewed as every published study. And there’s certainly no reason to assume that unpublished studies are more reliable than published studies.

    Furthermore, when it comes to THIS particular subject, there’s no reason to look at published vs. unpublished studies in the first place, there’s more than enough variation among published studies. There are published studies, especially if you focus on treating depression, that indicate the same results found in the unpublished studies. Basically all your doing by grabbing the the unpublished data is adding more negative results, so the question is: “Does that really tell you more about the effectiveness of psychotherapy, or is it just a mathematical artifact of adding more data?”

    At any rate we already know that psychotherapy for depression is a dodgy proposition because there are multiple causes of depression. The depressions most amenable to therapy are situational (like PTSD associated depression), but that’s a subset of depression cases. But we also know that most cases of situational depression resolve themselves because depression is a normal response to bad experiences.

    And even with situational depression the research isn’t terribly encouraging depending the type of therapy in vogue at the moment. Turns out for instance that therapy organized around a theory that suppressed emotions or repressed memories need to be unlocked or converted by repeatedly recounting the trauma; tend to prolong rather than resolve depression. For instance it’s NOT a good idea to flood a school with therapist who will try to make students recount the trauma of a shooting in the days and weeks immediately following event. Sometimes talking helps, sometimes it doesn’t.

    We already know that those suffering from Major Depression or Bi-Polar disorder get little additional benefit from therapy beyond somewhat minimized isolation and an additional set of eyes on the patient to flag decompensation.

    I even seem to recall a few studies that a couple hours of exercise physical actiity a week can do more than therapy to elevate mood and relieve depression.

    I guess I just see this as a study that’s pretending to tell us something we don’t already know and linking it to publication bias. I think it’s fine to be realistic about therapy and depression, but we don’t need bring publication bias into the conversation in order to do that. I wonder if bringing publication bias into the discourse might do more to obscure rather than clarify the subject?

    Nevertheless by original comment did miss the mark (and maybe this one does as well) in significant ways and I apologize for that. I thought about simply deleting it but that didn’t feel honest.

Leave a Reply