Early clinical trial results regarding new drugs and devices for the treatment of chronic medical conditions often turn out to be exaggerated, reports a new study by researchers at the Mayo Clinic in Rochester.
Indeed, more than a third of early clinical trials for drugs and devices in the study showed positive results that turned out to be substantially inflated when compared with the results from subsequent trials.
“Physicians and patients should be cautious about new or early clinical trial evidence,” said Dr. Fares Alahdab, one of the study’s authors and an evidence-based medicine researcher at the Mayo Foundation for Medical Education and Research, in a released statement. “Exaggerated results could lead to false hope as well as possibly harmful effects.”
The so-called Proteus phenomenon — the tendency of the positive findings of early clinical trials to be contradicted by later ones (named after the Greek sea-god who could quickly alter his appearance) — has been observed before, but primarily in the context of specific types of diseases, such as endocrine disorders. The Mayo Clinic researchers wanted to examine the phenomenon across a wide range of chronic illnesses.
Looking at the issue more broadly is important, given the large number of people currently living with one or more chronic medical conditions, such as heart disease, stroke, cancer, diabetes, kidney disease and chronic obstructive lung disease (COPD). Treatments for these conditions can save and extend lives, but they can also have significant side effects. Having accurate information to weigh the benefits and harms of new treatments is therefore essential.
Study details
For the study, Alahdab and his colleagues reviewed almost 2,700 randomized controlled clinical trials (considered the gold standard of research) that had evaluated a drug or device in patients with a chronic disease. All the studies had been published between January 2007 and June 2015 in one of the top 10 high-impact medical journals.
From that larger pool of studies, the Mayo researchers focused on 70 meta-analyses, which included the results of 930 clinical trials involving about 66,000 participants.
They found that 37 percent of the first and second studies published about a drug or device ended up with results that exaggerated the treatment’s positive effect on patients. On average, the treatment effect in the early studies was 2.67 times larger than what was eventually shown when subsequent trials were published.
Interestingly, the researchers found no association between the likelihood that a study would have exaggerated results and several factors known in general to influence study results, such as the study’s size and length, the risk of publication bias, and the source of the study’s funding.
The Proteus phenomenon “remains unpredictable,” Alahdab and his colleagues conclude, “and decision makers should deal with early evidence with caution.”
Not anti-innovation
“Some people may think this is an anti-innovation message,” said study co-author Dr. M. Hassan Murad, who heads the Mayo Clinic Evidence-Based Practice Research Program, in a released statement. “To the contrary, we welcome new treatments.”
“We just want people to know that the benefit seen in real practice, when treatments are given to people with various comorbidities and in different settings, may be smaller than what was seen in the earliest clinical trials,” he added.
In an accompanying editorial, Dr. Alex Krist, an assistant professor of family medicine and population health at Virginia Commonwealth University and a member of the U.S. Preventive Services Task Force, offers a similar warning.
“These findings demonstrate the need for caution when deciding whether to adopt early evidence,” he writes.
“The first study demonstrating benefit for a new intervention should most likely trigger further research rather than herald a significant practice change,” he adds.
Krist is addressing doctors, but patients should take heed as well.
FMI: The study was published online in the journal Mayo Clinic Proceedings, where it can be read in full. You can also read Krist’s editorial on the journal’s website.
