As Janet Raloff reported last week in Science News, a new study raises serious doubts about the reliability of studies published in Chinese-language academic medical journals.
The study, which was published in the May 25 issue of the open-access journal PLoS One, reviewed 369 meta-analyses published in Chinese journals since 1999. Meta-analyses are review articles that combine and compare data from many related studies to come up with a general hypothesis regarding them. They’re considered among the most authoritative types of research.
Almost 85 percent of those 369 review articles focused on traditional Chinese herbal remedies for various ailments. Most of the others examined the efficacy of acupuncture. A small percentage (about 1 percent) looked at the medical value of the centuries-old form of Chinese massage known as Tui na.
As Raloff notes, however, the 369 reviews “didn’t hold up very well” when compared to the scientific benchmarks that various organizations have established for doing a meta-analysis:
Many of the papers were incomplete, roughly one-third contained statistical errors and others provided data or comparisons that the authors termed misleading. Fewer than half of the surveyed papers described how the data they were presenting had been collected, how those data had been analyzed or how a decision had been made about which studies to compare. The majority of papers also did not assess the risk of bias across studies or offer any information on potential conflict-of-interest factors (such as who funded or otherwise offered support for the research being reviewed). …
Overall, “the quality of these reviews is troubling,” the [authors of the PLoS paper] conclude. … This is especially true, they point out, when considering that such peer-reviewed comparisons of therapies should be a key source of evidence-based medicine. Then again, they add, surveys of doctors and nurses in China have indicated that most “had not heard of or did not understand the meaning of evidence-based medicine.”
Interestingly, as the PLoS authors point out, Chinese researchers are increasingly publishing in international journals, where they must follow established evidence-based standards. (And, yes, I know very well that Western academic medical journals have their own problems with bias, conflicts-of-interest and other issues that undermine their findings. I’ve reported on that topic here many times.)
The PLoS authors reach no verdict about whether the medical treatments analyzed by these 369 meta-analyses are effective or not.
But, based on this study’s findings, no one should be citing Chinese-language studies as evidence that they are.
So, are the Chinese less healthy than we are as a result?