In the past, I’ve devoted at least a couple of Second Opinion columns (such as this one) to research that has suggested that bilingual speakers have a cognitive advantage, particularly in executive control (an umbrella term for “command and control” thinking processes, such as reasoning, planning and problem-solving).
I’ve also written in this space about publication bias in scientific journals — including how studies with positive results tend to get published, while those with negative results don’t.
Well, those two issues collided earlier this month with the online publication of a study in the journal Psychological Science. A team of researchers from the University of Edinburgh in Scotland and the University of Sassari in Italy — who themselves have published research that supports the idea that bilinguals have a cognitive advantage — have uncovered evidence that the literature on this topic may be skewed due to a bias toward publishing positive results.
This study does not mean that the idea of a bilingual cognitive advantage is false. But it does offer yet another important lesson to all of us about why we should always be skeptical about medical research.
The ‘file drawer effect’
The authors of the Psychological Science paper say they began to wonder if the claim of a bilingual cognitive advantage is an accurate reflection of all the research in the field when they realized that they, themselves, had submitted for publication only the results of experiments that had positive results. What if other researchers in the field had been doing something similar?
If that were the case, then the commonly accepted idea of bilingualism enhancing executive control may not be true. The advantage may simply be a reflection of a publication bias that favors positive results over negative or neutral ones.
This tendency of researchers to not submit their negative-result studies for publication is a well-known phenomenon known as the “file drawer effect.”
The researchers decided to see if they could find evidence of that effect in the research literature. They began by identifying all unpublished studies on bilingualism and executive control that had been presented at 169 relevant conferences around the world between 1999 and 2012. They found 104. Of those, 40 (38 percent) had results that supported a bilingual advantage, 14 (13 percent) had mixed results that appeared to support a bilingual advantage, 33 (32 percent) had mixed results that challenged the advantage (in other words, showed no executive control differences between monolinguals and bilinguals) and 17 (16 percent) found no advantage at all.
Half of those studies — 52 — were eventually published in 50 different journal articles. That number included 68 percent of the studies that clearly found a bilingual advantage and 50 percent of those that reported mixed-but-probably-positive results. By comparison, only 39 percent of the studies with mixed results that partly challenged the bilingual-advantage were published. The publication rate was even smaller — 29 percent — for the studies that found no differences between monolinguals and bilinguals or that reported a bilingual disadvantage.
“On the whole, 63% of the studies supporting the bilingual advantage were published compared with only 36% of the studies that challenged it,” the researchers write.
Further analysis revealed that the difference in publication rates could not be explained by the size of the study (how many participants were involved) or the specific cognitive tasks the participants were asked to do. The studies with findings supporting the bilingual advantage did tend to investigate fewer cognitive tasks, however, than those with mixed or negative results.
The authors of the Psychological Science paper were unable to determine why particular studies didn’t make it to publication. It may be, they say, that researchers are simply deciding on their own not to publish studies with mixed or negative results (the “file drawer effect”). Or, peer reviewers and editors may be rejecting studies that report those results more often than ones reporting positive results. (Studies with mixed or negative results are often judged to be more difficult to interpret and thus less worthy of publication.)
Staying open to the evidence
Although the published literature on bilingualism and executive control may not reflect all existing scientific evidence on the subject, that doesn’t mean bilingualism doesn’t confer a cognitive advantage, the authors of the current study are careful to point out.
“We agree that bilingualism should be conceived, a priori, as a positive and desirable achievement,” they write, “[but] we are also convinced that educational and political debates addressing the relevance of bilingualism should not be promoted by ignoring null or negative results.”
Their findings also underscore, they add, the ongoing need for the scientific community — no matter what the topic — to “be more open to studies that challenge the existing theories, especially when these are not yet fully established.”
“All data, not just selected data that supports a particular theory, should be shared,” the researchers conclude, “and this is especially true when it comes to data regarding issues that have enormous societal relevance and implications, such as bilingualism.”
You’ll find an abstract of the study on the Psychological Science website, but the study itself is behind a paywall.
