Skip to Content

Star Tribune shock poll: the methodology behind the Nolan-Cravaack outlier

In 2010, a KSTP/SurveyUSA poll showing Republican novice Chip Cravaack within a point of longtime DFL Congressman Jim Oberstar upended the 8th district race. How far has the GOP come in just one term? A new Star Tribune poll showing DFL challenger Rick Nolan besting Cravaack by 7 percentage points has produced hoots of incredulity.

No one truly knows where the electorate stands, but even Democrats find it hard to believe that Nolan leads Cravaack by more than Michele Bachmann leads Jim Graves. "This has been a neck-and-neck race and we believe it will continue to be close going into Election Day," Nolan’s campaign manager told the Strib’s Kevin Diaz.

Just because a poll seems like an outlier doesn’t mean it is. Many people mocked the KSTP/SurveyUSA poll in 2010, yet it proved prescient. SUSA has a strong track record in the state, and their most recent survey a week ago has Nolan up by 1 point, comporting with conventional wisdom and insanely high ad spending.

On the surface, you’d think the Strib poll, conducted by Texas-based Pulse Opinion Research, might be better. POR used a relatively large 1,000-person sample — nearly twice as big as SUSA’s — reducing mathematical sampling error to plus or minus 3 percentage points, compared to SUSA’s 4.2.

However, there are some methodological red flags that give this polling junkie pause.

First, POR polls on a single day — Tuesday, Oct. 16 in the Nolan-Cravaack case — compared to three for SUSA and four for Public Policy Polling (which also has a good Minnesota record).

Though pollsters refer to their results as a “snapshot in time,” single-day polling is not a best practice. It is too susceptible to a one day’s news or advertising. It doesn’t allow for callbacks, which some pollsters attempt to mitigate low response rates, perhaps the biggest trap door in this year’s numbers. If POR’s methodology box is gospel, they reached all 1,000 voters in just four hours.

Second, POR only calls land lines, even though a quarter of Minnesotans are cell-phone-only, according to Census Bureau estimates. POR says it augments landline respondents with cell phone panels, but Star Tribune digital editor Dennis McGrath says “that proved difficult at the [Congressional District] level, so it’s land lines only.”

Though there’s some evidence Democrats are underrepresented when cell phones are excluded, that might not be the case in the 8th. This is a split district, with a more DFL north (Duluth, Iron Range) and a GOP south (what I call the Emmer Exurbs). Cell phone coverage is spottier in the more remote north, so POR might have missed a lot of young Republicans.

The Strib’s published crosstabs did not include geography, so there’s no way for analysts to cross-check.

Then there’s using POR in the first place. Fivethirtyeight.com polling analyst Nate Silver gave Pulse Opinion Research remarkably low marks for the 2010 cycle, noting they “missed the final margin between the candidates by 5.8 points, a considerably higher figure than that achieved by most other pollsters.”

Pulse Opinion Research licenses methodology from Scott Rasmussen, whose polls aren’t considered as reliable as SUSA or PPP. Ras surveys often, but not always, have a Republican lean that is more pronounced the further away they are from Election Day, and very controversially weight for party I.D. That’s bad, because party I.D. changes — it’s an output, not an input — and weighting on something other than physical or geographic characteristics is risky.

However, the Strib may have mitigated POR's 2010 problems. McGrath notes, “This was our poll, not a Rasmussen Reports. No partisan weighting in these polls, at our direction.”

The Strib’s 8th District poll produced 7 points more Democrats than Republicans, while independent “partisan voter indexes” figure it +1 or +2 D. This is ripe for a GOP attack, but partisan I.D. can fluctuate and several pollsters with good records I’ve spoken with say not to sweat 4-5 point differences.

Still, the 8th District poll thrusts the Strib back into a familiar controversy: that its polls are too favorable to Democrats. Ironically, the paper switched pollsters this year — to former PiPress/KARE pollster Mason-Dixon. The Strib has not only stopped being outlier, its results have become leading indicators: they were the first to show the voting  amendment fight tightening, quickly ratified by SUSA and PPP.

This doesn’t mean the Strib is right — the three pollsters might be lemmings skittering toward a cliff. But in a cycle where two of the most dubious election surveys — from the University of Minnesota and St. Cloud State – have gone silent, it’s remarkable how little polling controversy there’s been until the Strib chose to deploy a new pollster and methodology two weeks before Election Day.

As hinted at by editor Nancy Barnes a month ago, this is the Strib’s first move into “robo-polling,” where respondents type answers to pre-recorded questions on their phone keypad. This is not beyond the pale — SUSA and PPP are robos, too. But again, both poll on more days, and SUSA only robo-dials land lines, hand-calling cell phones due to federal rules.

Why did the Strib go this route? Well, numbers addicts like me have been clamoring for independent House-race numbers, and POR is apparently pretty inexpensive. According to their rate card, a survey like this costs between $2,250 and $4,000. It's only a snapshot in time, but it will be interesting to see if Minnesota’s largest newspaper got what it paid for.

Get MinnPost's top stories in your inbox

Related Tags:

About the Author:

Comments (13)

Sounds about right to me

Cravaack's been an invisible Congressman. There's no reason why his constituents would favor a do-nothing invisible Congressman when they ousted a highly accomplished invisible Congressman just two years ago.

It might be right, but ...

Everyone I've talked to on the operative side has internal polling saying this is a 1-2 point race. People don't realize how much more GOP the 8th has become. The presence of Emmer Exurbs seem generally underrespected by urban lefties.

turnout

Different constituents will turn out in a presidential election. I believe this election will be won by whichever side turns out the most supporters. I also agree that the district leans Democratic but not +7.

Sounds like Barbra Streisand to me

Cravaack's been invisible? That's the best you can do? I wrote this article to highlight what Chip's done:

http://www.examiner.com/article/iron-range-s-biggest-newspaper-endorses-...

Here's what people who actually know what they're talking about said about Chip:

No longer do 8th District residents have to get patted on their heads and hear “things are being done behind the scenes” on these issues, which was Oberstar’s style. We are all now a party to the debate and discussion, and that is most welcome indeed.

The KSTP-SurveyUSA poll dramatically oversamples DFL voters. (They have a D+7 sample.) When you adjust it to a D+1 or +2 sample, Chip actually leads by almost 5 points, 48.8% for Chip to 44.1% for Nolan.

Polls

Not even I, as committed a DFLer as there ever has been, take Strib polls seriously at this point. Their money would be better spent paying interns.

Mason-Dixon's reputation

In May, the pollster for Al Gore's presidential bid in 2000 and John Edwards' in 2004 and 2008, Harrison Hickman, took the stand in the federal criminal case against Edwards over alleged campaign finance violations.

He was under oath to tell the truth.

Reported Oct. 2 on the left-wing blog Politico was this account by Josh Gerstein of that testimony: Hickman acknowledged that on January 2, 2008, a day before the Iowa caucuses, he sent out a summary of nine post-Christmas Iowa polls showing Edwards in contention in the Hawkeye State. However, he testified two-thirds of them were from firms he considered "ones we typically would not put a lot of credence in." Hickman put MASON-DIXON, Strategic Vision, Insider Advantage, Zogby and Research 2000 in the "less reputable" group. (Not included,by the way, was Rasmussen, which was rated by Fordham University as tied with Pew for the most accurate pollster in 2008.)

Ouch! Wondered who at the Star-Tribune selected Mason-Dixon? But if the liberal Hickman's assessment is correct, clearly the Minnesota Poll's history of erroneously predicting Democratic victories will be maintained.

You said Rasmussen "was rated

You said Rasmussen "was rated by Fordham University as tied with Pew for the most accurate pollster in 2008"

Only in the outdated initial Fordham report which was compiled in the hours immediately following the election, and was based on an ESTIMATE of Obama winning by a 6.15 margin. Obama, in fact, won by a 7.3 margin. Fordham released a FINAL report, based on the official election outcome. In it, Rasmussen & Pew were beat out by eight other pollsters.

http://www.fordham.edu/images/academics/graduate_schools/gsas/elections_...

More Than Fordham had Rasmussen No. 1

http://electoralmap.net/2012/2008_election.php

2008 Pollster Report Card which was the final analysis of 2008 polling had Rasmussen Reports
a clear No. 1.

Click on the "local mirror"

Click on the "local mirror" link in the "Data" section of that Report Card. The data used is long outdated information from November 12th, before all the ballots had been officially tallied. The Report Card has Obama winning by a spread of only 6.5 when, in fact, the spread was 7.3. It has Obama winning 52.6 to 46.1 when, in fact, Obama won 52.9 to 45.6. See page 5 of the official FEC election report for 2008 at http://www.fec.gov/pubrec/fe2008/federalelections2008.pdf

Fact: Obama won with a 7.3 spread, 52.9 to 45.6
Fact: Ipsos & CNN came closest with a 7-point spread, 53/46
Fact: Rasmussen underestimated Obama with a 6-point spread, 52/46
Fact: Rasmussen was not the most accurate pollster in 2008

Left wing blog Politico????

Wow, how far right do you have to be to make that kind of statement?

Long on speculation short of facts...

....I see a lot of opinion, but few cold hard facts supporting them. I do like the acknowledge then ignore tactic used in this piece, though. Maybe I have to be a "polling junkie" to understand all this, eh? For instance one day polling may be effected by news of the day or whatever: What day was it conducted? Not knowing means that that could cut both ways. Secondly I hear Democrats are using cell phones these days, shocking I know. (This is where the acknowledge then ignore comes in.) Another thing I hear is that the younger generation, the supposedly missed cell phone users, are supporting Obama. In fact the LA times tells me that they're supporting him 55% to Romney's 36%. Could it be that missing all those cell users could have cut the other way as well?

Really, what does this even mean: "Cell phone coverage is spottier in the more remote north, so POR might have missed a lot of young Republicans."

Again my failings as a polling junkie maybe the culprit, but if the Strib's polling favors Democrats it'd be the ONLY thing at that paper that does. I guess the lurch to the Right isn't 100% complete yet, eh?

Like everyone else, I have no idea if this poll is accurate or not. The writer and I have that in common.

Facts and opinon

Henk - the polling date (Oct. 16) is in the piece.

Dems may be undercounted nationally in polls that ignore cell phones, but that doesn't speak much to the 8th's particularities. My larger points are 1) when cell phones aren't called, we'll never know how cell-phone-only respondents feel and 2) don't assume national trends speak to the vagaries of the 8th.

We can never know if a poll is accurate, but there are best practices that reduce uncertainty. Whether in the name of cost or speed, the methodology used increases the doubt, especially when partisans trumpet a surprising result.  My post is a caution against crowing too loudly.

Emmer exoburbonites

... who want commuter and light rail connections to the cities.