Gallup hates the New York Times’ political polling analyst Nate Silver for the same reason the newspaper business hates Google. Both are aggregators that are undermining established business models. (It also might have something to do with Gallup’s lousy performance in the 2012 election cycle — more on that later.)
But let’s focus on aggregation, a word that’s frequently heard in the media business. Simply put, the Internet is full of people and websites that repurpose the original work of others.
A reporter for the Star Tribune or the New York Times spends many hours researching and writing a story. Then, as soon as it’s posted online, others snatch that original product and put their own spin on it to draw traffic — and ad dollars — to their own sites.
The aggregators get the benefit of the original work without having to pay for it. The Huffington Post, the Drudge Report, Gawker — these successful sites all got their start as aggregators, taking the work of others and adding their own (often snarky) spin to the original information. They collect an audience based on the work of others, and often profit handsomely from it.
Some of these sites have added original content and paid staff. But their operations are skeletal, compared with the news organizations that originate most of the content they feature.
Now comes Silver, who got his start as a baseball stats geek and parlayed it into a perch as the New York Times’ resident polling analyst. Silver looks at every publicly available poll and adds his own special sauce, adjusting for the varying methodologies of the individual pollsters.
For example, some pollsters use real people to ask the poll questions while some use robo-calling. Some call cell phones, some only call landlines. Silver considers these and numerous other factors, then comes up with his own predictions based on an aggregate of all the available polling data.
In case you haven’t heard, Silver called every state correctly in the presidential election and accurately predicted exactly how many electoral votes Barack Obama and Mitt Romney would each receive. He also called every Senate race correctly except for Heidi Heitkamp’s upset win in North Dakota.
The granddaddy of American pollsters, the Gallup organization, didn’t do as well. In fact, Silver ranked Gallup dead last — by a wide margin — among 23 polling organizations active in the 2012 election cycle. Now Gallup has fired back, ripping Silver (though not by name) for being — wait for it — an aggregator.
In a recent posting on the company’s website, Gallup Editor-in-Chief Frank Newport warned that Silver’s success could undermine the very polls that he relies on.
“It’s not easy nor cheap to conduct traditional random sample polls,” Newport wrote. “It’s much easier, cheaper, and mostly less risky to focus on aggregating and analyzing others’ polls.
“Organizations that traditionally go to the expense and effort to conduct individual polls could, in theory, decide to put their efforts into aggregation and statistical analyses of other people’s polls in the next election cycle and cut out their own polling. If many organizations make this seemingly rational decision, we could quickly be in a situation in which there are fewer and fewer polls left to aggregate and put into statistical models.”
It’s the same cri de coeur that has come from the traditional media in recent years: If we do the work, and others piggyback on it, eventually there won’t be anyone left to do the work in the first place.
The media business hasn’t figured out how to beat the aggregators. We’ll see if the pollsters can.
Did Nate Silver’s numbers really surprise anyone? His system is primarily an extension of baseball stats. Note, however, that the Gallup chief said polls are a “snapshot in time” rather than a prediction of the actual result; this is a wonderful get-out since there is no way of proving Gallup wrong on that basis.
Gallup’s likely voter screen
seemed to be the source of most of the error. “Conventional wisdom” is that likely voters are more republican by 2-3 points — Gallup had a 5-6 point shift — possible, surely, but very unlikely in a presidential year. And, now, with the news from Anonymous about how they foiled Rove’s (umpteenth) election-stealing plan, one has to wonder if Gallup was in on the fix, and tilting the numbers to give Rove and Romney reasonable deniability cover.
Maybe that’s not a bad thing
One principle in science is that you need to take care when measuring a property that the act of making that measurement doesn’t actually affect what the result will be. Or that you should at least be prepared to account for that possibility in your results.
I’ve long been uneasy with what seems to be the growing abundance of “polling” by what seems to be an increasing number of organizations – not all of which are as neutral as they really ought to be.
The peril – as I see it – is that members of the American public hear about some polling results that are skewing heavily in some candidate’s favor and then decide not to bother to vote since “One more won’t matter”. I’m not sure if anyone has ever studied the extent to which this occurs, but I have to believe that it does.
I’d like to see less polling. Or at least, that the polling being done is less suspect as to how “objectively neutral” it is.
That last one is the hardest to guard against, and frankly, I tend to think that Nate’s “third party analysis” (to the extent that he can continue to also remain objectively neutral) serves as a useful reality check on the possibility of poll skew.
So I like what Nate is doing. I think he “keeps them honest”, and hopefully he will be able to stand firm against the powers of big money and political influence for a good long time to come.
Intellectual property
The question I have is “Does what Nate Silver does add value?” For myself, whenever I get raw data, which is what pollsters provide, I want to know what it means. Pollsters, and those who cite their work have always seemed to assume their work was self explanatory, rarely in need of context, and never in need of analysis. Arguably, at least, Silver has taken raw data and used it in such a way that adds value, is indeed necessary if we are to derive meaning from that raw data. This seems to me an entirely useful function that lifts it above the role of aggregator, or the purveyor of snark.
With respect to Gallup, Silver is in the role of Consumer Reports testing the quality of work done by others, and forming opinions based on such tests which all are free to dispute. I never really thought of what Consumer Reports does as a violation of anyone’s intellectual property or some sort of improper aggregation of the work it reviews.
Aggregators
We have a similar issue with indexed mutual funds and ETFs. If everybody is buying indexes, and indeed indexes are comprised of indexes which are comprised of indexes, who is spending the money to analyze the stocks and determine the underlying value?
The article excerpt below describes much of the content and the business model of the MinnPost, right down to the snark (see Lambert’s comments at the end of nearly every item in the Daily Glean).
—————————
The aggregators get the benefit of the original work without having to pay for it. The Huffington Post, the Drudge Report, Gawker — these successful sites all got their start as aggregators, taking the work of others and adding their own (often snarky) spin to the original information. They collect an audience based on the work of others, and often profit handsomely from it.
who is spending the money to
who is spending the money to analyze the stocks and determine the underlying value?
And the suspicion is that stock analysis is worthless. That’s true with a lot of superficial analysis of polls, the kind you get from the conventional wisdom purveyors in the Washington based media. In any event, the proper analogy should be that the polls provide raw data. In the stock market, the analysts analyze the data and tell you what it means, buy sell or hold. Where polls are concerned, Silver and there are lots of others including all the usual cable media talking heads serve the role of analysts. Whether they are worth relying on is for each of us to judge.
Who Silver hurts
The folks who should be worried about the Nate Silver phenomenon aren’t the pollsters. He is not their competitor, he is their critic. And while some polling organizations may have been hurt by his negative reviews, others have been helped and might very well see their business pick up. No, the folks Silver threatens are the people who are in the same business he is, commentators on raw data provided by others. In other words, columnists, reporters, tv commentators. Many of those were visibly angered by Silver the upstart and quite properly so, because his precisions analysis was showing them up.
For years, if not centuries, we have all been complaining about the fact that we cover elections like horse races. The sometimes stated, often implied complaint about his, is that in doing so we are not giving sufficient consideration to the issues, giving voters the information they need to reach informed voting decisions. A criticism less often heard is that the fact is, we are covering the horse race side of elections very badly. We have seen that famed and esteemed commentators know virtually nothing about what they claim is the area of their expertise; the electoral politics of the United States.
This isn’t aggregation in the traditional sense
Aggregators merely repeat what others have said, and their add-on analysis is rarely if ever of any value at all. And, never do I hear of an aggregator who promises a truly comprehensive review of all statements on a particular topic.
Silver is taking other parties’ public statements of fact, and doing a great deal of analysis in order to determine how we (if we believe his methods) should think of those other parties’ statements. Then, he doesn’t do it randomly, but attempts to cover all of the field of pollsters so that he can speak of the known universe of polls. That’s much different than mere aggregation.
So, on that basis, I beg to differ from the good gentleman from Gallup’s smear of what Silver does.
And, regardless, shouldn’t we as a consuming public, and the public policy in general, reward and encourage analysis that actually leads to truth? At the end of the day, Silver was right, just as he was in the past. Sounds like a lousy reason to punish him.
People calling Silver an “aggregator” don’t understand statistical modeling. Silver does not just calculate a raw average, thus turning 10 smaller polls into one larger poll– he also adjusts the poll’s relative weight based on methodology, sample size, historical partisan slant, and others. He also includes structural predictors of the race, that have nothing to do with these polls– unemployment, GDP, consumer confidence, etc. This is only “aggregation” in the sense that someone who studies an issue closely and produces a succinct summary has “aggregated” a lot of raw data and turned it into knowledge useful if put towards a particular question.
Ultimately, the biggest lesson I take from this is that more statistics courses need to be taught in our high schools and colleges, so that more people understand that there is more to this than a big calculator.