[Updated at 5:20 to note likely voters polled via interviews, not a phone-keypad survey. A 6:30 update notes all adults polled for general election; likely voters for primary.]
Polls are catnip to political kitties, but Wednesday’s MPR/Humphrey Institute gubernatorial survey had political and journalism insiders coughing up hairballs this morning.
The figure that jumped out at everyone wasn’t 10 (Mark Dayton’s lead over Margaret Anderson Kelliher among likely DFL primary voters), 6 (Matt Entenza’s number), 4 (Dayton’s general-election lead over Tom Emmer) or 2 (Emmer’s lead over Kelliher and Entenza). Instead, it was 5.8 — the margin of sampling error, in percentage points plus or minus.
That’s a big MOSE! How big? Bigger than any poll Wednesday morning in Pollster.com’s 64-survey national compendium.
Margin of sampling error represents how smaller, sampled populations may differ from actual ones. You sometimes hear about a “95 percent” confidence level, which means there’s a 19 in 20 chance that the range (poll result plus or minus sampling error) represents all voters’ preference.
In this case, Dayton’s 35 percent general-election support among adults translates to a range of 29.2 percent (minus 5.8) to 40.8 percent (plus). Likewise, Emmer’s 31 percent support ranges from 25.2 percent to 36.8 percent. Because Emmer’s high is above Dayton’s low, the DFLer’s polling lead is said to be (well) within the margin of sampling error.
And there are even higher MOSEs when we get to the primary results.
Dayton’s lead over Kelliher and Entenza is built on a smaller subset of DFL voters, so the MOSE goes up — to 8.75 percent! Therefore, Dayton ranges between 29 and 47 percent, while Kelliher is between 19 and 37 percent. Even Dayton’s double-digit lead is within the margin of sampling error.
I know of at least one local journalism organization that didn’t report the results because the MOSE was so high.
I talked (for the second time in two days) with U prof. Larry Jacobs, who oversaw the survey. He makes a couple of points about judging this particular MOSE:
First, the comparisons between the MPR/HHH poll and other surveys likely isn’t apples-to-apples. Many, if not most, pollsters use a simple MOSE calculation based on sample size; Jacobs named Gallup and SurveyUSA. For a 700-person survey, this usually translates to around 4 percentage points, plus or minus.
However, Jacobs adds in his survey’s “design effects” — for example, the response rate. In this case, 26 percent of those called actually answer the questions. That’s a problem every pollster faces, and it increases the chance a sample isn’t random. That hikes the MOSE.
“We’re leaning over backwards to err on the side of caution,” Jacobs says of his higher reported MOSE. “We’re being over-earnest academics. If we used the margin of error [calculation] used by the industry, we wouldn’t be having this conversation.”
OK, fair enough, but still, 5.8 or 8.75 percentage points produces a huge range. How seriously should we take these numbers, professor?
“I wouldn’t over-read the results,” Jacobs says. “By no means are we saying Dayton has a large lead, or an insurmountable lead. Our analysis suggests that he does have a lead, and the statistics support that. But let’s not go the other way and suggest the results are somehow questionable.”
In other words, you’d rather have Dayton’s number than Kelliher’s, but don’t go crazy about the specific number. Of course not! Politicians and journalists would never do that!
However, MPR did refer to Dayton’s primary lead as “comfortable.” The story doesn’t reference MOSE until the 14th paragraph, and then only notes the DFL-Emmer matchups fall inside the sampling error range, even though that’s true for the Dayton-Kelliher result, too.
KARE11’s Scott Goldberg gets MOSE in his third paragraph, but also doesn’t note the primary-election caveat. (And the MOSE is likely even higher for another figure MPR and KARE reference: the results among DFL women — an even smaller subgroup where Dayton holds a lead.) The Strib’s brief doesn’t even mention MOSE.
Also, the highly quotable Jacobs can be his own worst enemy. For example, in MPR’s story, Jacobs termed the Dayton-Kelliher results “a real slap in the face to the Democratic Party,” since delegates just endorsed Kelliher. That’s defensible on some level — you’d expect an endorsee to be closer, if not the leader — except that Kelliher might have a lead, given the MOSE demonstration above.
Jacobs acknowledges a Kelliher lead is “possible, given the nature of probability statistics —but it’s not likely.”
In other words, in the range of possibilities, it’s more possible that Dayton’s out front. “These point estimates are not meaningless,” he stresses.
I’ll give Jacobs points for being more forthright about his higher MOSE calculation, but that means the public and journalists need to be more circumspect. If honesty produces higher MOSEs, then caveats need to creep higher into stories.
I know these are only polls, most of us aren’t math majors, and there are only so many details we can absorb. It’s fine to report who’s ahead, but headlines pronouncing “so-and-so leads the race”— when those numbers might vary substantially — demands a modification to “so-and-so leads the poll.”
Jacobs says he’ll include a fuller explanation of MOSE with future results, and he says the figure may drop as the election nears and the surveys are refined.
“All this is education,” he says. “If you go back 10 years ago, we were having long conversations about why there weren’t equal numbers of Republicans and Democrats in the survey. I hope we’ve made it clear to people that partisan ID is a function of the political system, and [the number] goes back and forth. With margin of error, there’s been kind of a habit on how to report it that hasn’t kept up with the times, and the result is a little bit of complacency.”