Disclaimer: What follows is extreme political and statistic nerdiness, with a major angels-dancing-on-head-of-pins factor and a huge trap door at the end. However, if you want to know how to correctly interpret a poll’s “margin of sampling error” — and I’m looking at some of you, political media — read on.
When a political poll is released, journalists check whether a lead is within the “margin of error.” If it is, a candidate’s advantage might be described as “narrow” or even “a statistical tie.” If the lead is outside the margin — a.k.a. “statistically significant” — the adjectives turn glowing, and the phrase “front-runner” enters the conversation.
But what if some of us figure this wrong?
Last week, I had to correct a couple of reporters and a political blogger on whether a Mark Dayton lead was outside the margin of error. I was right; it wasn’t.
But in getting the big question right, my method was wrong. After getting help from a mathematician, a political scientist and a pollster, I discovered doing it right means a few more leads are significant — especially if the races involve three or more serious candidates … like Minnesota’s for the last 15 years.
Yes, these are only polls, stat-heads like me spend too much time yammering about them, and we all need more issues and investigations in our media diet. But poll stories aren’t going away, so we might as well get them right.
The math: This won’t hurt, I think
Here’s a basic illustration from the most recent SurveyUSA/KSTP poll:
Dayton: 42 percent
Emmer: 37 percent
Margin of sampling error (henceforth known as error margin, for simplicity’s sake): plus or minus 3.7 percentage points
The most basic error: looking at the 3.7 and not the plus/minus. Some will see Dayton’s 5-point lead, note that it’s larger than 3.7, and pronounce statistical significance.
No. Dayton’s support could be 42 minus 3.7 (38.3 percent) and Emmer’s 37 plus 3.7 (40.3 percent). Because Emmer could be leading, people often write that Dayton’s advantage is “within the margin of error.”
This was my mistake.
Error margins refer to points: in this case, the 42 percent, or 37 percent. However, we’re usually more interested in leads: comparing two points. That’s calculated with a related, but different formula.
This is not a back-of-the-envelope calculation. So to help — and with the help of University of Minnesota math prof Larry Gray and Washington University Political Science Prof Steven Smith — I ginned up a handy-dandy spreadsheet.
Don’t fear, numbers-challenged citizens! You only need to know three numbers and enter them in the spreadsheet’s green cells:
1. The leading candidate’s support
2. The second-place candidate’s support
3. The number of poll respondents
A lot of other numbers will dance on the spreadsheet, but the answer: “Is the lead statistically significant?” will pop out in the orange field. (By the way, the numbers on the sheet are there for illustration; they’re to be replaced.)
Some pollsters hike their error margin by including “design effects” — uncertainty that their particular survey introduces. This number is a multiplier. If you don’t know design effects, it’s 1. If you can get the figure, its 1 plus some fraction.
Because pollsters generally don’t publish this figure, only the media sponsor may know it. But many news organizations include an email address if you have questions; can’t hurt to ask.
The three-candidate effect
Fundamental principles pop out when you understand the numbers.
First: Smaller leads are likelier to be statistically significant. I’ll spare you the math, but if you play with the spreadsheet and compare it to the “plus-minus” method, you’ll see.
Second: Smaller leads are more likely to be significant in a three-way race. Basically, #3’s share shrinks the possible variation in #1’s lead over #2.
Third: smaller leads are even more likely to be significant as #3’s support rises. (More variation shrinkage.)
By the way, these principles apply if there are four, five or an infinite number of candidates in the race. Conceptually, they all get treated like #3.
The angels-on-the-head-of-pin factor
OK, so does all this OCD add up to anything meaningful? Are there polling leads reported as significant when they’re not?
Gotta be honest: haven’t found any recent ones, though the latest Strib poll produced an almost-perfect example.
In it, Mark Dayton a 7-point lead with a 3.9-point plus or minus error margin. If you just doubled the error margin (to 7.8), Dayton’s 7-point lead isn’t statistically significant. But according to the Strib’s pollster, Princeton Survey Research Associates, the error margin of the lead comes within 0.03 points of significance … about as close as you can get to “outside the margin” without being so.
Rachel Stassen-Berger’s story (wisely) stayed away from significance or lack thereof, though it did provide the error margin of a point, not the lead.
Of course, the biggest question is whether “statistical significance” equals “real-world significance.” And here the trap door opens.
Political junkies know the litany of polling reliability issues: increasing refuse-to-respond rates, pollsters excluding cell phone users who’ve dropped their landlines, etc. These all potentially threaten significance more than math errors.
But the U’s Larry Gray notes a more fundamental problem.
Three-way races may produce more statistically significant leads, but voters’ preferences are less reliable. Instead of an either/or choice, voters face strategic considerations: If I’m a Democrat or Republican and vote for an independent, am I helping the other side win?
Even though pollsters often remind us that their results are “only a snapshot,” the three-way snapshot is blurrier — not due to math error, or even survey design, but simple human nature. As is often the case, the spreadsheet is more logical than the human heart.