Nonprofit, independent journalism. Supported by readers.


Star Tribune shock poll: the methodology behind the Nolan-Cravaack outlier

In a race even dueling operatives say is within a couple of points, a plus-7 Nolan result raises eyebrows.

In 2010, a KSTP/SurveyUSA poll showing Republican novice Chip Cravaack within a point of longtime DFL Congressman Jim Oberstar upended the 8th district race. How far has the GOP come in just one term? A new Star Tribune poll showing DFL challenger Rick Nolan besting Cravaack by 7 percentage points has produced hoots of incredulity.

No one truly knows where the electorate stands, but even Democrats find it hard to believe that Nolan leads Cravaack by more than Michele Bachmann leads Jim Graves. “This has been a neck-and-neck race and we believe it will continue to be close going into Election Day,” Nolan’s campaign manager told the Strib’s Kevin Diaz.

Just because a poll seems like an outlier doesn’t mean it is. Many people mocked the KSTP/SurveyUSA poll in 2010, yet it proved prescient. SUSA has a strong track record in the state, and their most recent survey a week ago has Nolan up by 1 point, comporting with conventional wisdom and insanely high ad spending.

On the surface, you’d think the Strib poll, conducted by Texas-based Pulse Opinion Research, might be better. POR used a relatively large 1,000-person sample — nearly twice as big as SUSA’s — reducing mathematical sampling error to plus or minus 3 percentage points, compared to SUSA’s 4.2.

Article continues after advertisement

However, there are some methodological red flags that give this polling junkie pause.

First, POR polls on a single day — Tuesday, Oct. 16 in the Nolan-Cravaack case — compared to three for SUSA and four for Public Policy Polling (which also has a good Minnesota record).

Though pollsters refer to their results as a “snapshot in time,” single-day polling is not a best practice. It is too susceptible to a one day’s news or advertising. It doesn’t allow for callbacks, which some pollsters attempt to mitigate low response rates, perhaps the biggest trap door in this year’s numbers. If POR’s methodology box is gospel, they reached all 1,000 voters in just four hours.

Second, POR only calls land lines, even though a quarter of Minnesotans are cell-phone-only, according to Census Bureau estimates. POR says it augments landline respondents with cell phone panels, but Star Tribune digital editor Dennis McGrath says “that proved difficult at the [Congressional District] level, so it’s land lines only.”

Though there’s some evidence Democrats are underrepresented when cell phones are excluded, that might not be the case in the 8th. This is a split district, with a more DFL north (Duluth, Iron Range) and a GOP south (what I call the Emmer Exurbs). Cell phone coverage is spottier in the more remote north, so POR might have missed a lot of young Republicans.

The Strib’s published crosstabs did not include geography, so there’s no way for analysts to cross-check.

Then there’s using POR in the first place. polling analyst Nate Silver gave Pulse Opinion Research remarkably low marks for the 2010 cycle, noting they “missed the final margin between the candidates by 5.8 points, a considerably higher figure than that achieved by most other pollsters.”

Pulse Opinion Research licenses methodology from Scott Rasmussen, whose polls aren’t considered as reliable as SUSA or PPP. Ras surveys often, but not always, have a Republican lean that is more pronounced the further away they are from Election Day, and very controversially weight for party I.D. That’s bad, because party I.D. changes — it’s an output, not an input — and weighting on something other than physical or geographic characteristics is risky.

However, the Strib may have mitigated POR’s 2010 problems. McGrath notes, “This was our poll, not a Rasmussen Reports. No partisan weighting in these polls, at our direction.”

Article continues after advertisement

The Strib’s 8th District poll produced 7 points more Democrats than Republicans, while independent “partisan voter indexes” figure it +1 or +2 D. This is ripe for a GOP attack, but partisan I.D. can fluctuate and several pollsters with good records I’ve spoken with say not to sweat 4-5 point differences.

Still, the 8th District poll thrusts the Strib back into a familiar controversy: that its polls are too favorable to Democrats. Ironically, the paper switched pollsters this year — to former PiPress/KARE pollster Mason-Dixon. The Strib has not only stopped being outlier, its results have become leading indicators: they were the first to show the voting  amendment fight tightening, quickly ratified by SUSA and PPP.

This doesn’t mean the Strib is right — the three pollsters might be lemmings skittering toward a cliff. But in a cycle where two of the most dubious election surveys — from the University of Minnesota and St. Cloud State – have gone silent, it’s remarkable how little polling controversy there’s been until the Strib chose to deploy a new pollster and methodology two weeks before Election Day.

As hinted at by editor Nancy Barnes a month ago, this is the Strib’s first move into “robo-polling,” where respondents type answers to pre-recorded questions on their phone keypad. This is not beyond the pale — SUSA and PPP are robos, too. But again, both poll on more days, and SUSA only robo-dials land lines, hand-calling cell phones due to federal rules.

Why did the Strib go this route? Well, numbers addicts like me have been clamoring for independent House-race numbers, and POR is apparently pretty inexpensive. According to their rate card, a survey like this costs between $2,250 and $4,000. It’s only a snapshot in time, but it will be interesting to see if Minnesota’s largest newspaper got what it paid for.