Nonprofit, nonpartisan journalism. Supported by readers.


Marriage amendment support soars, but how ‘likely’ is that?

The bump in support for the marriage amendment in a recent KSTP/SurveryUSA poll may represent a change in public opinion, but it may also reflect peculiarities of the poll itself.

Though it’s only one poll, on a tricky-to-poll topic, KSTP/SurveyUSA‘s 52-37 percent majority for the marriage amendment was a startler because other surveys (including earlier SUSA polls) showed the amendment in trouble.

Like them or not, polls can influence fundraising and volunteer recruitment, so reporters and citizens should know how they work. On Monday, our own Beth Hawkins debriefed local pollster Bill Morris about why KSTP’s poll is (for now) an outlier. Simultaneously, I was chatting up SUSA pollster Jay Leve for his take.

‘Apples and oranges’

Jay LeveCC/Flickr/debJay Leve

Leve surprised me with some media criticism about how KSTP reported aspects of the poll, which included questions about national and state races. “KSTP reported the Obama lead was ‘cut in half‘ since [our] last survey. We did not write that and we did not make that claim because the samples are different. It’s apples and oranges,” Leve stated.

This is a major point. From May 2011 through February 2012, respondents were good to go if they were registered to vote. Last week’s sample was different: SUSA deployed what’s commonly known as a “likely voter screen.” Voters had three options: “certain to vote,” “likely to vote,” or “unlikely to vote.” Only those who answered “certain to vote” were given the issue questions.

(Maybe we should call SUSA’s a “certain voter screen.”)

Though it might seem like the screen altered the results — “certain” voters as bigger amendment supporters than “registered” voters — there’s really no way to know. Lots of other factors, including shifting mood and normal statistical quirks, could be the culprits.

Weighting and accuracy

Still, any time a pollster introduces a screen or weighting (as SUSA does, matching Census figures for gender, age, region and cell phone use), they open themselves up to criticism.

Many amendment opponents noted 50 percent of 18-to-34-year-old voters supported the amendment, when previous polls show most opposed. Leve, who makes no apologies for the data, notes that young voters were still the least favorable age group in the new survey. And when it comes to the overall sample, Leve notes that SUSA’s final pre-election poll on the North Carolina marriage amendment (showing a +20 point margin for) was nearly the final result (+22 for).

Many Democrats — suspicious of KSTP’s GOP-funding owner Stanley Hubbard — instinctively suspect SUSA puts its thumb on the scale. I’ve reviewed their polls repeatedly over the years and their pre-election polls are among the most accurate. pollster/analyst Nate Silver also rates them near the top.

Polls this far out lack an imminent election as a fact-check, and Silver has suggested that likely voter screens before Labor Day generally “have not been any more accurate before that date. In fact, they have been slightly less accurate than registered voter polls.”

Public Policy Polling — whose June survey showing the amendment losing was the most recent before SUSA’s newest — uses a different screen. PPP polling director Tom Jensen says respondents need only answer they voted in one of the last three general elections (2006, 2008, 2010) with no questions (yet) about 2012.

“It’s somewhere between registered voters and likely voters,” he says.

Texting surveys?

Another methodological wedge issue: how the nation’s growing number of cell-phone-only voters are contacted. Automated* pollsters can’t legally call cell-phones, so PPP doesn’t.

Tom JensenTom Jensen

“We find that weighting our polls for age produces a good representation of the electorate even without calling cell phones,” Jensen says. “SurveyUSA is not calling cell phones; it is sending them a text survey on mobile devices, and it’s not clear whether this is going to be an effective way of surveying that population or not — they have not done it in previous election cycles.”

Leve again points to this cycle’s North Carolina results, which used the new method. SUSA’s Minnesota poll featured 26 percent cell-phone respondents, roughly in line with the most recent national Census data. The poll showed cell-phone-onlies (who tend to be younger and more liberal) opposing the amendment 48-39 while landline respondents supported it 57-34.

Fundamental question

One of the most obvious differences is in the question asked. Since May 2011, SUSA’s has been:

“An amendment to the Minnesota Constitution on the ballot defines marriage as between one man and one woman. Will you vote FOR the amendment? Against the amendment? Or not vote on the measure?”

PPP’s, which almost exactly duplicates the actual ballot wording (without the disputatious title):

“Should the Minnesota Constitution be amended to provide that only a union of one man and one woman shall be valid or recognized as a marriage in Minnesota?”

Leve says SUSA began polling on the question in 2011, before the ballot language was determined, and has stuck with the wording for consistency. However, given that the samples changed with the most recent poll, the question probably should have, too. 

[* – corrected from original story. Human operators can call cell phones, but so-called “robo” pollsters like PPP can’t use auto-dialing. They could opt for human dialing, but that is usually more expensive.]

Comments (7)

  1. Submitted by Paul Udstrand on 07/27/2012 - 09:10 am.

    Lack of credibility

    Seems to me that KSTP/SurveyUSA team has established a record of bogus results haven’t they?

    • Submitted by Beth Hawkins on 07/27/2012 - 01:26 pm.

      Not so much

      I checked into this, too, albeit through different channels than David. And I found the same thing: SurveyUSA typically comes up with reliable, unbiased results. I do think the cellphone/texting wrinkle is an interesting place to dig further.

  2. Submitted by Ray Schoch on 07/27/2012 - 09:54 am.

    A poll is a poll is a poll

    Methodology aside, and most of us in the general public aren’t qualified to evaluate the statistical methods of how the generalizations are arrived at, the only poll that counts won’t occur until November. This poll may simply reflect methodological errors, or, as David noted, perhaps the amendment’s opponents’ suspicions that the Republican-leaning funder of the station has his thumb on the scale, so to speak, are justified. Or, it certainly seems possible that this poll, different from others though it may be, is entirely accurate, and the proposed amendment will be adopted.

    Should that be the case, it wouldn’t be the first time that bigotry has been enshrined in a constitution. After all, we already have it similarly enshrined in a statute. Odd how that “small government = less intrusive government = better government” rhetoric on the right disappears when it comes to the sexual.

  3. Submitted by Steve Titterud on 07/27/2012 - 01:52 pm.

    Oh, REALLY ?

    “We find that weighting our polls for age produces a good representation of the electorate even without calling cell phones,” Jensen says.

    How, EXACTLY, did you find that? I mean, how do you contact the cell phone users, who are “illegal” to call, to find out how they WOULD have answered IF they’d been asked ? And then, how do you translate THAT into putative survey results ??

    I am extremely doubtful of the glib assertion above, “We find…”.

  4. Submitted by Kenneth Kjer on 07/27/2012 - 04:15 pm.


    I find it interesting that I have not been able to find one person who opposes this amendment that was included in the poll or has ever been polled about the amendment.

    • Submitted by Max Hailperin on 07/28/2012 - 09:59 am.

      Look at the sample size.

      The article you are commenting on shows that the question about the amendment was asked of a grand total of 552 individuals throughout the state. It would be a miracle if you could find any of them, regardless where they stand on the issue.

  5. Submitted by Max Hailperin on 07/28/2012 - 10:19 am.

    Many 18-34 year olds aren’t registered

    I’m curious about the impact of election-day registration. Assuming that the 2012 election is like the 2008 one, something like a quarter of the 18-34 year olds who vote won’t be registered even the day before the election, let alone months before. And I am not aware of any reason to think that this quarter is statistically similar to the ones who are registered now (or registered now and say they are certain to vote). Before Survey USA even gets to their “likely voter” screen, they already have narrowed their pool down to just registered voters.

Leave a Reply