In recent months, MinnPost’s David Brauer and I have both posted stories questioning the accuracy of several different public opinion polls by supposedly reliable firms that reported surprising and contradictory numbers.

In June, I reported that the Democratic-leaning Public Policy Polling showed a 10-point shift in opposition to the proposed constitutional amendment banning same-sex marriage. Barack Obama had just voiced his support for gay marriage, so that bump made sense.  

The following month, Brauer and I wrote separate pieces probing two SurveyUSA polls about the proposed amendment, one in May and the other in July, that showed a 20-point swing in the other direction. Perplexed, we each called on different experts who said the numbers seemed odd to them.

In my story, Minneapolis public opinion expert Bill Morris, principal of Decision Resources, said that the crosstabs — the data subsets breaking things down by age, political affiliation and so forth — seemed completely wrong to him. Way too few young people said they’d be voting no, for example, and the standard gender gap was missing.

Brauer’s story quoted the pollster saying he disagreed with the media’s take-away from his survey, and broke down the quirks in its methodology.

Brauer took another detailed look at the topic in mid-September, when the two firms again released numbers: “SUSA has the anti-gay-marriage amendment claiming a majority even without undecideds, 50-43, while PPP posited a nailbiter, 48-47,” he wrote.

A fascinating look at polling chaos

What gives? A recent New York magazine story offers a fascinating, compulsively readable feature on the chaos plaguing polling, “The. Polls. Have. Stopped. Making. Any. Sense.” That headline being the 46-character tweet sent out by polling wunderkind Nate Silver after a post-Democratic National Convention poll showed Obama beating Mitt Romney decisively in scarlet Wisconsin and Romney besting Obama in New Hampshire.

Silver, of course, is the onetime baseball-stat geek whose uncanny knack for calling elections dead-on got his blog, FiveThirtyEight, picked up by the New York Times. He has a new book out, “The Signal and the Noise: Why So Many Predictions Fail.”

The New York piece is truly worth a read. Its gist, which isn’t done justice by condensing here, is that while Americans are increasingly hungry for poll data, fewer and fewer media outlets are paying for polls, leaving the field to pollsters with skin in the game.

‘Polling’s dark age’

At the same time, conventional polling methodologies are increasingly outdated, to the point where the percent of potential voters contacted who actually agree to answer poll questions may be in the single digits. And without reliable mechanisms for compensating, outcomes are ever more dependent on “weighting,” or the pollster’s statistical adjustment for variables.

To wit:

“The rising demand for trustworthy polling analysis also reflects something disturbing about the data itself. The central problem is that prototypically modern science is being disrupted by new technologies, which have created a flood of new firms and new methods. ‘We’re in sort of what I would call polling’s dark age,’ says Jay Leve, who runs the polling firm Survey USA. ‘We’re coming out of a period of time where everyone agreed about the right way to conduct research, and we’re entering into a time where no one can agree what the right way to conduct research is.’”

And the release of a poll can have an immediate effect on a candidate or campaign, particularly late in the game when undecideds may find it tempting to join what looks to be the winning side.

Will this October’s “surprises” be polls skewed to affect the outcomes of various contests? Stay tuned.