Does excluding cell phones give Dayton-Emmer-Horner polls the wrong number?

Last week, the University of Minnesota’s Humphrey Institute and MPR released a gubernatorial poll showing DFLer Mark Dayton and Republican Tom Emmer deadlocked with 34 percent each. It was the first survey in months that didn’t have Emmer trailing.

As disappointed partisans are wont to do, some DFLers seized on what they said was a methodological failing: the poll only included voters with landlines, excluding those who’ve gone cell phone-only. The implication: if surveys included voters who’d “cut the cord,” Dayton would likely be the leader.

Polls are political catnip, not governance. Still, they theoretically influence everything from fundraising to voter attitudes, and media outlets spend tens of thousands of dollars on them. According to a May 2010 Pew Research Center report, 23 percent of adults (and 25 percent of households) are now cell-only. Fully 49 percent of 25- to 29-year-olds have cell phones and no landlines.

So why exclude them from your survey?

“The main reason for not including cell phones in our sample is cost,” says Joanne Miller, an associate professor of political science who works on the HHH survey.

Unlike many politicians, Miller is up-front about the trade-offs. She says reaching a cell-only user costs twice as much as a landline. A major reason: the federal Telemarketing and Consumer Fraud Abuse Act says cell phone calls must be manually dialed — no computer dialers banging out random digits, as with landlines.

And even as you spend more to dial, you’re less likely to reach an eligible voter. Miller notes cell phones aren’t likely to be shared, and a “significant number” are in the hands of minors. That means more re-dialing, and paying more.

“We chose to do two other things that cost money: have a longer survey, which enables us to get more nuance, and bump our sample size [from 500 or so] to 750,” Miller says. “That allows us to make stronger conclusions about smaller percentage differences.”

This might seem a bit torturous, but Miller says the bottom line is that cell phone exclusion doesn’t move the polling needle all that much: “The demographic differences are big, but to be honest, there are very few political differences” that affect the ultimate results.

One reason: cell phone-only users are far less likely to vote than landline-users. Cell users are younger and less white, two groups that tend to vote less and comprise less of the adult population.

An illustration: The Pew survey (actually a series of relatively massive surveys of over 1,500 respondents each conducted between January and April) asked about party identification. After grouping hardcores and leaners, cell-phone-only users tilted Democratic 52-35. The landline sample? 45-41.

The result if you combined the two groups? 47-40. 

It’s a small difference — plus-7 Dem compared to plus-4 Dem. However, Miller says given the large sample size, could be statistically significant. That gives some credence to Democratic criticisms.

The Pew poll also showed Obama job approval/disapproval is 45-45 in the landline sample versus 47-42 when cell phones were added. In the Congressional horserace, Dems were down 6 points among landliners but even when cells were tossed in.

(Most polls show Dem numbers deteriorating since Pew’s spring meta-survey.)

Even though they eschew spending the coin, landline-only pollsters don’t totally ignore cell phone users. Miller says HHH’s model assigns a greater weight to responses from voters who have landlines and cell phones. That’s an attempt to mimic the cell-only part of the population.

However, trade-offs again: While a larger sample size reduces her margin of sampling error, additional weighing bumps it up. The most recent survey’s was 5.3 percentage points, plus or minus.

At least one local media outlet has bought its way out of the problem. In August, the Star Tribune bragged that its Minnesota Poll — which showed Dayton with a 10-point lead — dialed cell phones as well as landlines.

“It did cost us more money,” says Star Tribune assistant managing editor Kate Parry, who oversees political coverage. “There was a general sense that this is a growing thing, it reached a tipping point … [W]e’ve gotten to the point that the cost is worth it.”

The Strib, which effectively emerged from bankruptcy a year ago, has increased spending in some coverage areas. The paper has not reconstituted its in-house polling operation, disbanded due to budget cuts in 2007. Since then, it uses New Jersey-based Princeton Survey Research, which also did the Pew meta-study.

The paper’s Aug. 1 gubernatorial survey received 831 responses, more than the HHH poll, and reported a margin of sampling error of 4.5 percentage points. (Some pollsters only include sample size when calculating margin of sampling error; the Humphrey Institute includes design effects such as weighting.)

For now, the Strib stands nearly alone in including cell-phone-only Minnesota voters.

Survey USA, which partners with KSTP-TV, has traditionally been landline-only; such a survey gave Dayton a 14-point margin over Emmer in early August. SUSA recently included cell phones in Washington and North Carolina contests.

(The Washington margin moved a single percentage point when cell-only respondents were added; in North Carolina, there was no change.)

Rasmussen Reports, which like SUSA uses robo-dialers and automated questions, is also landline-only. Its Aug. 13 poll showed Dayton up by 9 points. Fox9 is Rasmussen’s local media partner.

Decision Resources, Inc., a local firm which last released a poll in June that had Dayton up by 12, does include cell phones in its random digit dialing, a company official says.

You can also learn about all our free newsletter options.

Comments (7)

  1. Submitted by Jason Walker on 09/07/2010 - 11:15 am.

    Thanks, David, for this information. I haven’t had a land line in 10 years and none of my friends under 30 does, either.
    Every time I read a story about a poll I wonder about whether cell phones were called, and if not, I pretty much discount the results. And I completely agree that polls do influence voter attitudes, fairly or not.

  2. Submitted by Brian Simon on 09/07/2010 - 12:05 pm.

    In addition to the methodology question about cell vs landline dialing, I am somewhat curious about their method for determining whom are the likely voters. In national reporting we’re hearing a lot about GOP-leaning voter enthusiasm, and how that is likely to boost Repub chances of picking up seats, nationally. However, being above average here in Minnesota, we tend to have nation-leading turnout in elections. So… When Princeton Survey or other national pollsters apply their ‘likely voter’ algorithm to Minnesota Polls, do those algorithms reflect our typically high turnout, or do they use a ‘one size fits all’ system that perhaps erroneously excludes likely voters in Minnesota?

  3. Submitted by Matthew Steele on 09/07/2010 - 12:06 pm.

    It’s hard for me to believe that 51% of 25-29 voters actually have landlines. My guess would be less than 25%. And the ones I know who have landlines have them due to bundles with Comcast or Qwest, and many do not even have phones hooked up to the lines.

  4. Submitted by Bill Gleason on 09/07/2010 - 01:07 pm.

    This is a very interesting topic. I have an opinion, but since I have never owned a cell, my opinion on the matter is worthless…

    But the Strib may be in a position to provide some answers. How much do the results differ when cell phone users are compared with land line users?

    Does it make a diffence… or not?

  5. Submitted by rolf westgard on 09/07/2010 - 02:32 pm.

    This one has to be wide open. When you have both candidates as weak as these two, anything could happen. Horner could become very significant, but with an impact that’s hard to figure at this point.

  6. Submitted by Richard Schulze on 09/07/2010 - 09:33 pm.

    Projected Election Results:

    FiveThirtyEight Forecasts
    September 3rd, model

    Based on polling and demographic data, this is the prediction of how the state will vote on election day.

    Dayton 78.9 % chance of win
    Emmer 21.2 % chance of win

    Dayton 46.9
    Emmer 41.1
    Horner 9.5

    http://elections.nytimes.com/2010/forecasts/governor/minnesota

  7. Submitted by Bernice Vetsch on 09/08/2010 - 12:33 am.

    Since most telephone users of either type are also computer users, might on-line polls provide a more representative estimate of public opinion than telephone polls?

Leave a Reply