Are three-way-race polling leads more significant than we think?

Disclaimer: What follows is extreme political and statistic nerdiness, with a major angels-dancing-on-head-of-pins factor and a huge trap door at the end. However, if you want to know how to correctly interpret a poll’s “margin of sampling error” — and I’m looking at some of you, political media — read on.

When a political poll is released, journalists check whether a lead is within the “margin of error.” If it is, a candidate’s advantage might be described as “narrow” or even “a statistical tie.” If the lead is outside the margin — a.k.a. “statistically significant” — the adjectives turn glowing, and the phrase “front-runner” enters the conversation.

But what if some of us figure this wrong?

Last week, I had to correct a couple of reporters and a political blogger on whether a Mark Dayton lead was outside the margin of error. I was right; it wasn’t.

But in getting the big question right, my method was wrong. After getting help from a mathematician, a political scientist and a pollster, I discovered doing it right means a few more leads are significant — especially if the races involve three or more serious candidates … like Minnesota’s for the last 15 years.

Yes, these are only polls, stat-heads like me spend too much time yammering about them, and we all need more issues and investigations in our media diet. But poll stories aren’t going away, so we might as well get them right.

The math: This won’t hurt, I think
Here’s a basic illustration from the most recent SurveyUSA/KSTP poll:

Dayton: 42 percent
Emmer: 37 percent
Margin of sampling error (henceforth known as error margin, for simplicity’s sake): plus or minus 3.7 percentage points

The most basic error: looking at the 3.7 and not the plus/minus. Some will see Dayton’s 5-point lead, note that it’s larger than 3.7, and pronounce statistical significance.

No. Dayton’s support could be 42 minus 3.7 (38.3 percent) and Emmer’s 37 plus 3.7 (40.3 percent). Because Emmer could be leading, people often write that Dayton’s advantage is “within the margin of error.”

This was my mistake.

Error margins refer to points: in this case, the 42 percent, or 37 percent. However, we’re usually more interested in leads: comparing two points. That’s calculated with a related, but different formula.

This is not a back-of-the-envelope calculation. So to help — and with the help of University of Minnesota math prof Larry Gray and Washington University Political Science Prof Steven Smith — I ginned up a handy-dandy spreadsheet.

You can find it here.

Don’t fear, numbers-challenged citizens! You only need to know three numbers and enter them in the spreadsheet’s green cells:

1. The leading candidate’s support
2. The second-place candidate’s support
3. The number of poll respondents

A lot of other numbers will dance on the spreadsheet, but the answer: “Is the lead statistically significant?” will pop out in the orange field. (By the way, the numbers on the sheet are there for illustration; they’re to be replaced.)

Some pollsters hike their error margin by including “design effects” — uncertainty that their particular survey introduces. This number is a multiplier. If you don’t know design effects, it’s 1. If you can get the figure, its 1 plus some fraction.

Because pollsters generally don’t publish this figure, only the media sponsor may know it. But many news organizations include an email address if you have questions; can’t hurt to ask.

The three-candidate effect
Fundamental principles pop out when you understand the numbers.

First: Smaller leads are likelier to be statistically significant. I’ll spare you the math, but if you play with the spreadsheet and compare it to the “plus-minus” method, you’ll see.

Second: Smaller leads are more likely to be significant in a three-way race. Basically, #3’s share shrinks the possible variation in #1’s lead over #2.

Third: smaller leads are even more likely to be significant as #3’s support rises. (More variation shrinkage.)

By the way, these principles apply if there are four, five or an infinite number of candidates in the race. Conceptually, they all get treated like #3.

The angels-on-the-head-of-pin factor
OK, so does all this OCD add up to anything meaningful? Are there polling leads reported as significant when they’re not?

Gotta be honest: haven’t found any recent ones, though the latest Strib poll produced an almost-perfect example.

In it, Mark Dayton a 7-point lead with a 3.9-point plus or minus error margin. If you just doubled the error margin (to 7.8), Dayton’s 7-point lead isn’t statistically significant. But according to the Strib’s pollster, Princeton Survey Research Associates, the error margin of the lead comes within 0.03 points of significance … about as close as you can get to “outside the margin” without being so.

Rachel Stassen-Berger’s story (wisely) stayed away from significance or lack thereof, though it did provide the error margin of a point, not the lead.

Of course, the biggest question is whether “statistical significance” equals “real-world significance.” And here the trap door opens.

Political junkies know the litany of polling reliability issues: increasing refuse-to-respond rates, pollsters excluding cell phone users who’ve dropped their landlines, etc. These all potentially threaten significance more than math errors.

But the U’s Larry Gray notes a more fundamental problem.

Three-way races may produce more statistically significant leads, but voters’ preferences are less reliable. Instead of an either/or choice, voters face strategic considerations: If I’m a Democrat or Republican and vote for an independent, am I helping the other side win?

Even though pollsters often remind us that their results are “only a snapshot,” the three-way snapshot is blurrier — not due to math error, or even survey design, but simple human nature. As is often the case, the spreadsheet is more logical than the human heart.

Comments (10)

  1. Submitted by Matt Pettis on 10/27/2010 - 10:28 am.

    Appreciate the analysis… following along the spreadsheet, I think there’s an error in labeling that should be corrected just to make things clear: A9 is ‘p2 sq’, and I think it is meant to be ‘(p1 – p2) sq’… easy to tell from the formula, but the correction will make things clear.

  2. Submitted by Matt Pettis on 10/27/2010 - 10:34 am.

    No to be nit-picky, but the ‘A’ column labels seem to be very goofed up… they don’t correlate to the formula going on in the B column mostly, unless I’m misunderstanding that the ‘A#’ is relating to the values corresponding to the A# cells… I think it would help if that could get cleaned up a bit (otherwise, it really isn’t helpful). The C column comments, on the other hand, are a bit more helpful.

    Again, don’t mean to be nit-picky — this is really cool stuff, and fixing these labels can make things more understandable.

  3. Submitted by Bill Gleason on 10/27/2010 - 10:43 am.

    Nice analysis.

    As you point out the trick of doubling the +/- moe is not quite correct, but as we used to say, it is close enough for government work. Given other serious statistical problems, I don’t think I’m going to pull out Excel to analyze any further.

    The much more serious error is to forget to double the moe. Thus ten points looks like a big lead, but it isn’t so big if the moe is +/-6.

    I have to think about the third part effect. But getting a bead on that one is very difficult, too, as you point out. The “effective” moe on that one is probably grossly underestimated.

    Just the thoughts of a scientist who works with numbers a lot, not a statistician.

    [As an aside, I’ll note that accurate error estimates are extremely difficult to make in science. The error derived from statistics for a chemical bond length – determined experimentally by X-ray methods – is probably an OVERESTIMATE by at least a factor of 2-3x…]

  4. Submitted by David Brauer on 10/27/2010 - 10:52 am.

    Matt – thanks. The math is right; the column A labels were off because I moved all the green fields together.

    All fixed now. If you spot other labeling errors, let me know. Appreciate it.

  5. Submitted by Julie Blaha on 10/27/2010 - 11:08 am.

    How do undecided’s factor in here? Does that category make every race at least a three way consideration? Or would it make sense to reduce the number of people surveyed to factor them out?

  6. Submitted by Stephan Flister on 10/27/2010 - 11:19 am.

    Remembering too that the results are at some confidence level, usually 95%, meaning that of 100 such samples taken, 5 will likely be outside the quoted margin.

    Also, the results for each candidate are not independent. Say both candidates in a two way race get 50% in some poll with a 3% margin of error. If one candidate’s “real” score is 52% (+2 in the error margin) the other candidate’s “real” score cannot also be 52.

  7. Submitted by Adam Platt on 10/27/2010 - 01:29 pm.

    If we can put our slide-rules away for just a moment . . . I pay almost no attention to polls, but I do make note of them in a three-way race when I am considering voting for a third-party candidate. But when I read about polls, I always wonder to what extent coverage of polls (especially specious or sloppy coverage of polls) drives voter behavior. That’s when I start to care.

    I’m sure someone has looked at this, but I’m less interested in polling than their effects and when they raise or suppress voter participation, the latter being the most noxious effect of our horserace preoccupation with politics, something I am hardly immune from.

  8. Submitted by Dion Goldman on 10/27/2010 - 02:03 pm.

    Adam, I could not agree more. Polls and the online, TV & newspaper’s who publish them ARE swaying the voter. Those who stand to benefit from “horse race” type reporting, generates Millions $ in negative ads, which people hate, but oddly enough, let negative ads influence them; without being aware that MOST ads are a distortion of the candidates positions.

    The Public has no time (or lazy) to think for themselves, read & study candidate positions. That plus cynicism of the system. So they let an ad or sound bite, or the color of the candidates shoes sway their vote.

    And when I tell people to pay no attention to polls and explain why, they agree, but in the end they are swayed by polls. Very sad for democracy and our republic.

  9. Submitted by Hénock Gugsa on 10/27/2010 - 02:39 pm.

    @#8 -“…”horse race” type reporting, …”
    Call it the Dan Rather blather!

  10. Submitted by Bill Gleason on 10/27/2010 - 03:26 pm.

    Slide-rules? I haven’t used a slide-rule for 37 years…

    Polls are fun and I don’t think they do any harm. As long as they are reported honestly and very conservative claims made by the media as far as their meaning.

    If your candidate is “behind” then you should get out there and work like heck for him/her. By the same token if you are “ahead” you should be wary of how little this actually means and work like heck for your candidate.

    Finally, if you are considering voting for a third party candidate, and it is apparent that they don’t have a prayer, you have to ask yourself a question.

    Do I want to vote for my candidate as a matter of principle or do I want to vote for the second best – in my opinion – because the third best is even worse?

Leave a Reply