Because I’m a numbers geek, I’ve taken a closer at the bizarre divergence between two similarly timed U.S. Senate polls — one that put Norm Coleman 10 points ahead (KSTP/Survey USA) and another with Al Franken up by 9 (Star Tribune/Minnesota Poll).
Over the weekend, colleague Eric Black and I noted the stark differences in party ID; the Strib poll had 42 percent Democrats and just 26 percent Republicans; SUSA’s numbers were 37 and 30, respectively.
Today, I got a look at some deeper breakdowns, and “party ID gap” only explains about a quarter of the 19-point gap. Even if the Strib poll had used SUSA’s partisan breakdown, Franken would still have led — by 4 to 5 points, outside within the margin of error.
The reason? Partisans in the two polls react quite differently.
Here’s how Democrats split in the Strib poll:
Franken 78 percent
Coleman 6 percent
Barkley 12 percent
Now, Democrats in the SUSA poll:
Franken 65 percent
Coleman 10 percent
Barkley 19 percent
The Strib’s Republicans:
Coleman 78 percent
Franken 6 percent
Barkley 12 percent
Coleman 83 percent
Franken 6 percent
Barkley 10 percent
In other words, the Democrats in the Strib’s poll were far more loyal to Franken, while SUSA’s Republicans were somewhat more loyal to Coleman.
I find it hard to believe the Strib’s conclusion — that Dems are as enamored of Franken as Republicans are of Coleman. Few polls this season have shown that. However, events can change an election and old partisan behaviors can reassert, I guess.
Age, not geography
Still, the question remains: why would partisans react so differently?
Minnesota Independent’s Steve Perry has a theory: SUSA might have “an oversampling of Christian conservatives (in other words, outstate Minnesota)” while the Strib oversampled metro Democrats.
I asked the Strib’s pollster, Larry Hugick of Princeton Survey Research, about geographic disparities.
He said he defines the metro area a bit differently than SUSA does, but didn’t see significant variations regionally. (We didn’t talk religion, but Hugick notes pollsters do weight by age, race/ethnicity, gender and geography — including urban/rural splits — to reflect the latest Census figures.)
Hugick says the single most divergent factor between the two polls is age. Here, the numbers aren’t just different, they’re insanely different:
Franken 60 percent
Coleman 23 percent
Barkley 16 percent
Coleman 44 percent
Franken 33 percent
Barkley 13 percent
That’s right; Franken’s support among this group in the SUSA poll is half what it is in the Strib poll. Differing party ID shares, or the higher margin of error for subgroups, can’t explain that chasm’s enormity.
(I stupidly didn’t ask about cellphones, but will update if I find something out. Hugick says neither poll called cellphone users; if they had, Franken would probably benefit. Most stories I’ve read says the effect is a point or two at best, and even in younger groups, wouldn’t explain this.)
Polls were once similar
I asked Hugick about methodological differences with SUSA, beyond the major one: real people conduct his interviews while SUSA’s “robo-poll” makes you punch numbers into your phone after hearing recorded questions.
One divergence that leaped to mind, he said, is that he also weights his sample by education level — SUSA doesn’t. Most pollsters do try to make education reflect the relevant Census sample, he says.
In his piece, Perry made the not-illogical point that both polls might be outliers. It’s possible, though no one can know who’s right at the moment.
However, here’s another interesting datapoint: while the surveys sport a 19-point gap now, they were within 3 points of each other three weeks ago — the last time both measured the Senate race.
In fact, the Strib was more favorable to Coleman then — they had Norm up 4, while SUSA gave him a 1-point lead.
Panning for voters
I asked Hugick if, since the September survey, he’d changed his “likely voter” screen — the way pollsters down the stretch try to forecast who’ll actually cast a ballot. This requires figuring out intent — for example, if the respondent won’t vote even though he says he will.
Hugick explains that his “voter screen,” like many, gets more sophisticated down the stretch. Back in September, he basically used a two-question screen: “Will you definitely or probably vote?” and “Did you vote in 2004?”
Now, he says, he uses several questions to tease out intent, including whether folks voted before in their district, whether they voted in the 2006 governor’s race.
However, he noted the bottom line didn’t really change much: 87 percent of his sample passed through the screen this month, compared to 88 percent in September.
Final thought: Minnesota Poll critics and supporters hearken back to its track record, but I wouldn’t. This is the first election the Strib has outsourced the Minnesota Poll; before last year’s buyouts, it was all handled in house. (The Strib’s last employee poll director, Rob Daves, wrote about the tricky “likely voter” question here.)
While Hugick says he uses many similar methods to Daves’ Minnesota Poll, they aren’t identical. However, I’m not going to go that deep into the rabbit hole right now.