Nonprofit, nonpartisan journalism. Supported by readers.

Donate

A little more on the Strib’s wild U.S. Senate poll

Because I’m a numbers geek, I’ve taken a closer at the bizarre divergence between two similarly timed U.S. Senate polls — one that put Norm Coleman 10 points ahead (KSTP/Survey USA) and another with Al Franken up by 9 (Star Tribune/Minnesota Poll).

Over the weekend, colleague Eric Black and I noted the stark differences in party ID; the Strib poll had 42 percent Democrats and just 26 percent Republicans; SUSA’s numbers were 37 and 30, respectively.

Today, I got a look at some deeper breakdowns, and “party ID gap” only explains about a quarter of the 19-point gap. Even if the Strib poll had used SUSA’s partisan breakdown, Franken would still have led — by 4 to 5 points, outside within the margin of error.

The reason? Partisans in the two polls react quite differently.

Here’s how Democrats split in the Strib poll:
Franken 78 percent
Coleman 6 percent
Barkley 12 percent

Now, Democrats in the SUSA poll:
Franken 65 percent
Coleman 10 percent
Barkley 19 percent

The Strib’s Republicans:
Coleman 78 percent
Franken 6 percent
Barkley 12 percent

SUSA:
Coleman 83 percent
Franken 6 percent
Barkley 10 percent

In other words, the Democrats in the Strib’s poll were far more loyal to Franken, while SUSA’s Republicans were somewhat more loyal to Coleman.

I find it hard to believe the Strib’s conclusion — that Dems are as enamored of Franken as Republicans are of Coleman. Few polls this season have shown that. However, events can change an election and old partisan behaviors can reassert, I guess.

Age, not geography
Still, the question remains: why would partisans react so differently?

Minnesota Independent’s Steve Perry has a theory: SUSA might have “an oversampling of Christian conservatives (in other words, outstate Minnesota)” while the Strib oversampled metro Democrats.

I asked the Strib’s pollster, Larry Hugick of Princeton Survey Research, about geographic disparities.

He said he defines the metro area a bit differently than SUSA does, but didn’t see significant variations regionally. (We didn’t talk religion, but Hugick notes pollsters do weight by age, race/ethnicity, gender and geography — including urban/rural splits — to reflect the latest Census figures.)

Hugick says the single most divergent factor between the two polls is age. Here, the numbers aren’t just different, they’re insanely different:

Strib 18-to-34-year-olds:
Franken 60 percent
Coleman 23 percent
Barkley 16 percent

SUSA 18-to-34s:
Coleman 44 percent
Franken 33 percent
Barkley 13 percent

That’s right; Franken’s support among this group in the SUSA poll is half what it is in the Strib poll. Differing party ID shares, or the higher margin of error for subgroups, can’t explain that chasm’s enormity.

(I stupidly didn’t ask about cellphones, but will update if I find something out. Hugick says neither poll called cellphone users; if they had, Franken would probably benefit. Most stories I’ve read says the effect is a point or two at best, and even in younger groups, wouldn’t explain this.)

Polls were once similar
I asked Hugick about methodological differences with SUSA, beyond the major one: real people conduct his interviews while SUSA’s “robo-poll” makes you punch numbers into your phone after hearing recorded questions.

One divergence that leaped to mind, he said, is that he also weights his sample by education level — SUSA doesn’t.  Most pollsters do try to make education reflect the relevant Census sample, he says.

In his piece, Perry made the not-illogical point that both polls might be outliers. It’s possible, though no one can know who’s right at the moment.

However, here’s another interesting datapoint: while the surveys sport a 19-point gap now, they were within 3 points of each other three weeks ago — the last time both measured the Senate race.

In fact, the Strib was more favorable to Coleman then — they had Norm up 4, while SUSA gave him a 1-point lead.

Panning for voters
I asked Hugick if, since the September survey, he’d changed his “likely voter” screen — the way pollsters down the stretch try to forecast who’ll actually cast a ballot. This requires figuring out intent — for example, if the respondent won’t vote even though he says he will.

Hugick explains that his “voter screen,” like many, gets more sophisticated down the stretch. Back in September, he basically used a two-question screen: “Will you definitely or probably vote?” and “Did you vote in 2004?”

Now, he says, he uses several questions to tease out intent, including whether folks voted before in their district, whether they voted in the 2006 governor’s race.

However, he noted the bottom line didn’t really change much: 87 percent of his sample passed through the screen this month, compared to 88 percent in September.

Final thought: Minnesota Poll critics and supporters hearken back to its track record, but I wouldn’t. This is the first election the Strib has outsourced the Minnesota Poll; before last year’s buyouts, it was all handled in house. (The Strib’s last employee poll director, Rob Daves, wrote about the tricky “likely voter” question here.)

While Hugick says he uses many similar methods to Daves’ Minnesota Poll, they aren’t identical. However, I’m not going to go that deep into the rabbit hole right now.

You can also learn about all our free newsletter options.

Comments (10)

  1. Submitted by Sally Rolczynski on 10/06/2008 - 10:31 pm.

    Boring analysis David.

    If you want a real polling website, go to http://www.fivethirtyeight.com

  2. Submitted by David Brauer on 10/06/2008 - 10:57 pm.

    Aw Sally, have I acquired a regular first commenter?

    Can’t argue my way into the style points, but while I love Nate at 538, he didn’t have the crosstabs and thus, some of the analysis and explanation. That’s what I’m trying to add to the party here.

    The great thing about the Internet is one source does not corner the market.

  3. Submitted by John Olson on 10/07/2008 - 06:54 am.

    The voters of this state have been notorious ticket-splitters over the years. Consider the Strib/KMSP poll conducted between October 15-18, 1998, prior to the general election for Minnesota Governor:

    MINNESOTA GOVERNOR POLL Oct. 15-18

    Humphrey 35%
    Coleman 34%
    Ventura 21%

    Source: Minneapolis Star Tribune/KMSP-TV
    Margin of Error: +/- 3.5% pts

    Guess who won that race?

    Right now, the polling data means exactly zero. Zip. Nada. If Coleman and Franken (in particular) can’t get it out of the gutter, I wouldn’t discount Barkley’s chances.

  4. Submitted by David Brauer on 10/07/2008 - 08:56 am.

    John –

    Pollsters will tell you their numbers are snapshots in time, not necessarily (but certainly possibly) wrong given the ultimate results.

    I remember working for Newsweek in 1998 and telling them right about this time that Ventura could win. (Didn’t put any money on it, though.) It’s entirely likely Barkley could come up along the rail and push Franken to last – one of the Strib results I question the most is that Barkley is currently drawing equally from D’s and R’s – historically, IP guys take a bit out of the D’s as Jesse’s race showed.

    Again, all polls show is a scientific guess at this point – I wouldn’t call it exactly meaningless, but the power is in the voters’ hands. I fully expect Barkley’s numbers to climb and wouldn’t be shocked to see him win, not this year for sure.

    I do wonder, though, if someone will do an intelligent analysis of what his spending freeze and deficit-closing-in-a-downturn would mean to average Minnesotans. Not necessarily criticizing it, but I think the media often fails to vet IP policy prescriptions because they take a late-campaign shine to the “outsiderness.”

  5. Submitted by Brian Simon on 10/07/2008 - 01:26 pm.

    I too like 538, but Nate’s analysis of these polls stopped at pointing out the disparity in party ID, while Mr Brauer’s follow-through to analyze the age disparity is far more interesting.

    SUSA’s 44 for Coleman among 18 to 34 year olds sounds extremely implausible.

  6. Submitted by Reggie McGurt on 10/07/2008 - 03:57 pm.

    Excellent analysis. I’d also like to point out, as I have before on MinnPost, that SUSA’s robocalls have some very basic and fundamental methodological flaws. I’ve now gotten robocalled (does it work as a verb?) for SUSA’s senate race polls three separate times. All three times the republican/conservative response has been the first choice for EVERY single question. This seems likely to result in response bias, especially when you’re dealing with a push button poll.

    Another problem I encountered is that if you enter the wrong choice, you can’t do anything to fix it. After I accidentally picked the wrong guy, I just hung up. I have no idea if this was counted as a vote or not. Again, this seems likely to result in bias in favor of the first choice, which is always republican/conservative.

    Finally, I had to chuckle at John Olson’s comment that Franken in particular had to get out his campaign of the gutter. I don’t think either Coleman or Franken has run a clean campaign, but Coleman’s ads have reached new lows in substanceless mudslinging.

  7. Submitted by John Olson on 10/07/2008 - 04:46 pm.

    My comment was directed at both Coleman and Franken, Reggie. I can now see how easy it could have been misinterpreted. My bad.

    I will completely agree that Coleman’s ads are truly at bottomfeeder level.

  8. Submitted by Tom Anderson on 10/07/2008 - 07:06 pm.

    The pollsters now make a big point of the “snapshot in time” stuff, especially since they’ve been so far off in recent elections. There is often “the latest poll the day of or day before was accurate” used after the fact, but of course the results aren’t announced until after the vote. It is probably more likely that 20% of voters are flipping back and forth from day to day on who they are voting for…

  9. Submitted by Peter Frost on 10/08/2008 - 12:50 pm.

    David — Whose poll do you believe more accurately reflects the electorate this year? Here in VA, I’m a bit out of touch with the race, but it seems that the major parties have clear weaknesses this time around — Coleman as a GOP guy in a blue state tired of Bush, and Franken as a political newcomer with questionable credentials. Reminds me of the Ventura victory. But I don’t know if Barkley’s got the charisma and money to generate enough support to make a legit run at a third-party victory this time around…

  10. Submitted by David Brauer on 10/08/2008 - 02:34 pm.

    Tom –

    Polls really ARE snapshots in time, but the good pollsters release their down-to-the-wire tracking polls even after the election so you can see the arc. And yes, damn humans and their mind-changing ways do screw things up. (Things seem extra volatile this year, BTW.)

    Peter –

    I’m no more gifted at divining the future than the next slob, but I’d say a) Barkley winning wouldn’t shock me, especially this year and b) I don’t think the Strib poll is as much an outlier as others. It’s better designed than SUSA’s robo-poll. The Humphrey Poll sample is damn small.

Leave a Reply