Though it’s only one poll, on a tricky-to-poll topic, KSTP/SurveyUSA‘s 52-37 percent majority for the marriage amendment was a startler because other surveys (including earlier SUSA polls) showed the amendment in trouble.

Like them or not, polls can influence fundraising and volunteer recruitment, so reporters and citizens should know how they work. On Monday, our own Beth Hawkins debriefed local pollster Bill Morris about why KSTP’s poll is (for now) an outlier. Simultaneously, I was chatting up SUSA pollster Jay Leve for his take.

‘Apples and oranges’

CC/Flickr/debJay Leve

Leve surprised me with some media criticism about how KSTP reported aspects of the poll, which included questions about national and state races. “KSTP reported the Obama lead was ‘cut in half‘ since [our] last survey. We did not write that and we did not make that claim because the samples are different. It’s apples and oranges,” Leve stated.

This is a major point. From May 2011 through February 2012, respondents were good to go if they were registered to vote. Last week’s sample was different: SUSA deployed what’s commonly known as a “likely voter screen.” Voters had three options: “certain to vote,” “likely to vote,” or “unlikely to vote.” Only those who answered “certain to vote” were given the issue questions.

(Maybe we should call SUSA’s a “certain voter screen.”)

Though it might seem like the screen altered the results — “certain” voters as bigger amendment supporters than “registered” voters — there’s really no way to know. Lots of other factors, including shifting mood and normal statistical quirks, could be the culprits.

Weighting and accuracy

Still, any time a pollster introduces a screen or weighting (as SUSA does, matching Census figures for gender, age, region and cell phone use), they open themselves up to criticism.

Many amendment opponents noted 50 percent of 18-to-34-year-old voters supported the amendment, when previous polls show most opposed. Leve, who makes no apologies for the data, notes that young voters were still the least favorable age group in the new survey. And when it comes to the overall sample, Leve notes that SUSA’s final pre-election poll on the North Carolina marriage amendment (showing a +20 point margin for) was nearly the final result (+22 for).

Many Democrats — suspicious of KSTP’s GOP-funding owner Stanley Hubbard — instinctively suspect SUSA puts its thumb on the scale. I’ve reviewed their polls repeatedly over the years and their pre-election polls are among the most accurate. Fivethirtyeight.com pollster/analyst Nate Silver also rates them near the top.

Polls this far out lack an imminent election as a fact-check, and Silver has suggested that likely voter screens before Labor Day generally “have not been any more accurate before that date. In fact, they have been slightly less accurate than registered voter polls.”

Public Policy Polling — whose June survey showing the amendment losing was the most recent before SUSA’s newest — uses a different screen. PPP polling director Tom Jensen says respondents need only answer they voted in one of the last three general elections (2006, 2008, 2010) with no questions (yet) about 2012.

“It’s somewhere between registered voters and likely voters,” he says.

Texting surveys?

Another methodological wedge issue: how the nation’s growing number of cell-phone-only voters are contacted. Automated* pollsters can’t legally call cell-phones, so PPP doesn’t.

Tom Jensen

“We find that weighting our polls for age produces a good representation of the electorate even without calling cell phones,” Jensen says. “SurveyUSA is not calling cell phones; it is sending them a text survey on mobile devices, and it’s not clear whether this is going to be an effective way of surveying that population or not — they have not done it in previous election cycles.”

Leve again points to this cycle’s North Carolina results, which used the new method. SUSA’s Minnesota poll featured 26 percent cell-phone respondents, roughly in line with the most recent national Census data. The poll showed cell-phone-onlies (who tend to be younger and more liberal) opposing the amendment 48-39 while landline respondents supported it 57-34.

Fundamental question

One of the most obvious differences is in the question asked. Since May 2011, SUSA’s has been:

“An amendment to the Minnesota Constitution on the ballot defines marriage as between one man and one woman. Will you vote FOR the amendment? Against the amendment? Or not vote on the measure?”

PPP’s, which almost exactly duplicates the actual ballot wording (without the disputatious title):

“Should the Minnesota Constitution be amended to provide that only a union of one man and one woman shall be valid or recognized as a marriage in Minnesota?”

Leve says SUSA began polling on the question in 2011, before the ballot language was determined, and has stuck with the wording for consistency. However, given that the samples changed with the most recent poll, the question probably should have, too. 

[* – corrected from original story. Human operators can call cell phones, but so-called “robo” pollsters like PPP can’t use auto-dialing. They could opt for human dialing, but that is usually more expensive.]