Nonprofit, nonpartisan journalism. Supported by readers.


KSTP pollster explains excluding cell-phone-only voters

I’ve developed an Eric-Black-on-the-10th-Amendment fascination with cell phones and political polling, and have chatted up Minnesota’s gubernatorial pollsters here and here. And estimated 23 percent of adults live in “cell phone only” households — no landlines. Only the Star Tribune and Decision Resources, Inc. include cord-cutters in their surveys.

Now into the dock comes SurveyUSA president Jay Leve, whose company released the most recent gubernatorial poll, for KSTP.  DFLer Mark Dayton led Republican Tom Emmer 38 to 36 percent, with Independence candidate Tom Horner getting 18 percent. It nearly mirrored the MPR/Humphrey Institute’s 34-34-13 results days earlier. Neither poll included cell-phone-only respondents, henceforth known as CPOs.

CPO holdouts have two fundamental explanations: cost, and the relatively small electoral impact of CPOs, who tend to be younger and less politically involved. Leve offered both arguments, but with nuance — and a surprising bit of criticism for a fellow CPO-excluding pollster.

Because of various laws and demographics, the Humphrey Institute’s Joanne Miller told me earlier this month that CPOs costs twice as much to poll as landliners. Leve pegged the rise higher: “three- or four-fold.”

He says, “KSTP is to be commended as the only broadcaster in the Twin Cities who pays hard-cash dollars out of its pocket to commission opinion research. If SurveyUSA insisted that a pre-election poll include CPO respondents, there may not be any KSTP polling in Minnesota, as a practical matter.

“Then the question becomes: what is better, a survey of Minnesota without CPO respondents? Or, no survey of Minnesota? I don’t mean that as a rhetorical question. It’s a real question: Thoughtful persons can answer in different ways.”

The must-include-CPOs argument becomes stronger each passing year. In 2008, there were nearly half as many CPO households (13 percent) as there are today. That’s a big reason the Strib added CPOs this cycle.

For his station’s part, KSTP political reporter Tom Hauser says, “It’s something we’re considering in the future when we determine whether there’s a significant difference to justify the added cost.  It looks unlikely for this election cycle, but is something we would like consider more strongly in the future as more people become cell phone-only users.”

Ignoring a quarter of America’s households is only remotely justifiable if that cohort doesn’t move the needle much. Leve notes two SUSA surveys showed just that — in Washington State and North Carolina, where the company spent extra dough to include CPOs.

To his credit, Leve does not say this is a fixed principle: “I am not implying, David, that the results of an election poll will always be the same with CPO respondents included, nor even that the results would be the same a majority of the time.”

This brings us back to the moral dilemma of polling 80 percent of the public instead of 95 percent. CPO-exclusion critics have tended to be Democrats, since CPOs have polled more blue than the general population. A pre-primary Strib poll, for example, showed Mark Dayton’s lead jumping from 5 percent to 10 when CPOs were added in.

Here, Leve cautions CPO-exclusion critics from extrapolating his results. “There is no way to know, in advance, whether or how the results of a poll of Minnesota would be different, had a pollster systematically and defensibly included CPO respondents.”

Leve concluded with surprisingly harsh words for the Humphrey Institute’s method of closing the CPO hole. Miller told me her MPR polls add additional weight to respondents who have landlines and cell phones. She explains, “To the extent that people with have a landline AND a cell phone are similar to the people who just have a cellphone … then can help a little bit (how much so, we are not sure) … but it’s not likely to hurt.”

Leve says this sort of hope is wrong-headed. “There is no defensible weighting that can be done to a universe of reachable respondents [landliners] to compensate for a universe of unreachable respondents [CPOs].  There is a 50% chance that weighting, in an attempt to compensate for respondents who had zero chance of being included, will make the data worse, not better. ”

In sum, he says, “SurveyUSA acknowledges that there are limitations to telephone research in 2010.”

By the way, pollster and analyst Nate Silver of places SUSA near the top of his June 2010 pollster rankings. The Star Tribune was middle of the pack, while the Humphrey Institute ranked near the bottom. Silver’s rankings use polls back to 1998 — well before cell phones were a factor — but weighs recent polls more heavily.

You can also learn about all our free newsletter options.

Comments (10)

  1. Anonymous Submitted by Anonymous on 09/21/2010 - 11:44 am.

    Well – my wife and I are anything but “less politically involved,” or “younger,” for that matter, but we have no land line. Anecdotal to be sure, but I’m betting we’re not alone in our phone preferences.

  2. Submitted by Lora Jones on 09/21/2010 - 12:22 pm.

    In another string, the large percentage of the yougest voters (18-34 if I remember correctly) choosing Emmer was noted (48%). I did wonder at the time whether the exclusion of cell phones might have contributed, although I suspect that the small sample size had an even larger effect.

  3. Submitted by Michael Hunt on 09/21/2010 - 01:05 pm.

    I suspect a few days before the election you’ll be writing:

    “KSTP pollster explains excluding Democratic voters”

  4. Submitted by Rick Prescott on 09/21/2010 - 02:27 pm.

    “Because of various laws and demographics, the Humphrey Institute’s Joanne Miller told me earlier this month that CPOs costs twice as much to poll as landliners. Leve pegged the rise higher: ‘three- or four-fold.'”

    I’m sure these people know their industry better than I do, but this is just crazy. There are costs associated with polling, and the price of reaching someone can’t reasonably be used to decide who is included or excluded.

    The fact is, systematically excluding 25% of the population from any survey for any reason will lead to less reliable results. No amount of weighting based on assumptions (tested or not) can make up for a deficiency that large, and no adjustment to the margin of error figure makes the numbers any more useful.

    Put plainly, polls which exclude CPOs are significantly less useful to everyone. No satisfying argument can be made to the contrary.

    One would think that the industry would be motivated to overcome this anachronistic behavior. If not the pollsters themselves, surely their clients (and the public) should be demanding maximum accuracy.

  5. Submitted by Steve Titterud on 09/21/2010 - 03:46 pm.

    Who would pay good money for a poll which excluded so many? Who would buy the argument that it doesn’t really matter? Someone is selling snake-oil here.

  6. Submitted by Matthew Steele on 09/21/2010 - 03:55 pm.

    Um, I don’t know anyone under 30 with a landline. And we vote.

  7. Submitted by Steven Liesch on 09/22/2010 - 05:11 am.

    As a member of the “newspaper” generation,I gave up a “land-line” years ago and my wired children are all voters.

    I guess the only polling that will matter soon will be push polling.

  8. Submitted by karl karlson on 09/22/2010 - 08:22 am.

    And there are those of us who lie to pollsters either in person or when they call at dinner time.

  9. Submitted by Ray Marshall on 09/22/2010 - 06:08 pm.

    Isn’t the fact that most/many people refuse to answer calls from unknown callers by using Caller ID a bigger problem for pollsters than excluding cell phone only owners?

  10. Submitted by Howard Miller on 09/25/2010 - 07:27 pm.

    I’m glad we don’t run elections – the polls that matter – based on whether it is just too expensive to have some of our citizens ovote … like our neighbors in rural parts of the state, people who can’t make it physically to the polling place so we print mail-in ballots ….

    when your polling method systematically excludes important parts of the population you hope to generalize about, buyer beware!

Leave a Reply