Nate Silver Loses His Shit At HuffPost’s Ryan Grim

-686Days -15Hours -39Minuts -36Seconds

If you remember 2012 (I’d just as soon not, for personal reasons), the hot GOP issue was “unskewing the polls.” Republicans claimed that pollsters were simply missing millions of conservative voters, and that when all was said and done, Mitt Romney was going to win pretty much every state.

That didn’t go too well for the GOP in 2012.

Now, as the 2016 election comes down the home stretch, the accusation of”unskewing” is being flung around by rival pollsters,and it heated up to boiling yesterday.

One of the things we’re seeing is a strong belief that Hillary Clinton will win. Some of the polling analysts, as I’ve noted for a couple of days now, are projecting a virtually certain win for Clinton, as high as 98-99%.

The outlier here is everyone’s nerd crush from 2012, Nate Silver, who gives Trump a heartstopping 35% or so chance of winning. 

What’s the discrepancy? According to Ryan Grim of the Huffington Post, Silver is monkeying with the poll results, both nationally and at the state level.

There is one outlier, however, that is causing waves of panic among Democrats around the country, and injecting Trump backers with the hope that their guy might pull this thing off after all. Nate Silver’s 538 model is giving Donald Trump a heart-stopping 35 percent chance of winning as of this weekend.

He ratcheted the panic up to 11 on Friday with his latest forecast, tweeting out, “Trump is about 3 points behind Clinton ― and 3-point polling errors happen pretty often.”
So who’s right?

The beauty here is that we won’t have to wait long to find out. But let’s lay out now why we think we’re right and 538 is wrong. Or, at least, why they’re doing it wrong. 
The short version is that Silver is changing the results of polls to fit where he thinks the polls truly are, rather than simply entering the poll numbers into his model and crunching them.
Silver calls this unskewing a “trend line adjustment.” He compares a poll to previous polls conducted by the same polling firm, makes a series of assumptions, runs a regression analysis, and gets a new poll number. That’s the number he sticks in his model ― not the original number.
He may end up being right, but he’s just guessing. A “trend line adjustment” is merely political punditry dressed up as sophisticated mathematical modeling.
Guess who benefits from the unskewing?
By the time he’s done adjusting the “trend line,” Clinton has lost 0.2 points and Trump has gained 1.7 points. An adjustment of below 2 points may not seem like much, but it’s enough to throw off his entire forecast, taking a comfortable 4.6 point Clinton lead and making it look like a nail-biter.
It’s enough to close the gap between the two candidates to below 3 points, which allows Silver to say that it’s now anybody’s ballgame, because “3-point polling errors happen pretty often.”

Grim goes on to make several other points, which make a great deal of sense to me. Warning: I’m an amateur here. Consider the source. To wit:

For the polls to be wrong, there wouldn’t need to be one single 3-point error. All of the polls ― all of them, as Brianna Keilar would put it ― would have to be off by 3 points in the same direction. That’s happened before, but in 2012 the error favored President Barack Obama. In 2014, it favored Republicans. Errors are just as likely to favor Clinton as they are to favor Trump, and they would have to favor Trump. And we still haven’t accounted for the unique fact that one campaign has a get-out-the-vote operation, while the other doesn’t.

By monkeying around with the numbers like this, Silver is making a mockery of the very forecasting industry that he popularized. “The idea that she’s a prohibitive, 95 percent-plus favorite is hard to square with polling that has frequently shown 5- or 6-point swings within the span of a couple weeks, given that she only leads by 3 points or so now,” he told Politico recently. “[E]verything depends on one’s assumptions, but I think that our assumptions ― a Clinton lead, sure, but high uncertainty ― has repeatedly been validated by the evidence we’ve seen over the course of the past several months.”
I get why Silver wants to hedge. It’s not easy to sit here and tell you that Clinton has a 98 percent chance of winning. Everything inside us screams out that life is too full of uncertainty, that being so sure is just a fantasy. But that’s what the numbers say. What is the point of all the data entry, all the math, all the modeling, if when the moment of truth comes we throw our hands up and say, hey, anything can happen. If that’s how we feel, let’s scrap the entire political forecasting industry.
Silver’s guess that the race is up for grabs might be a completely reasonable assertion ― but it’s the stuff of punditry, not mathematical forecasting.
Punditry has been Silver’s go-to move this election cycle, and it hasn’t served him well. He repeatedly pronounced that Trump had a close to 0 percent chance of winning the Republican primary, even as he led in the polls. “Trump’s chances [are] higher than 0 but (considerably) less than 20 percent,” he wrote in November.

Grim also notes that Silver isn’t just screwing with national poll results, but state level surveys too.

So if Nate Cohn and Nate Silver both see a roughly 3-point race, why is one Nate confident in a Clinton win and the other sparking a collective global freakout?

Because Silver is also unskewing state polls, which explains, for instance, why 538 is predicting Trump will win Florida, even as we and others (and the early vote) see it as a comfortable Clinton lead. To see how it works in action, take the Marist College poll conducted Oct. 25-26. Silver rates Marist as an “A” pollster, and they found Clinton with a 1-point lead. Silver then “adjusted” it to make it a 3-point Trump lead. HuffPost Pollster, meanwhile, has near certainty Clinton is leading in Florida. 

Perhaps there’s a good justification for what Silver is doing. I think it’s bet-hedging, personally. He got pilloried for blowing the Trump nomination. But as Grim points out, the answer to too much punditry in data analysis is not more punditry – it’s less. Stick to the data and unless there’s some empirical reason to do otherwise, trust the data.

We don’t know, however, what Silver’s rationale is, because instead of offering it he freaked out on Grim in an epic tweetstorm that sheds virtually no light on the subject. Politico:

It began with “This article is so fucking idiotic and irresponsible,” and got only somewhat more polite from there.

Nate Silver unloaded Saturday on the Huffington Post’s Ryan Grim, who accused the polling guru and founder of the prediction website fivethirtyeight.com of “changing the results of polls to fit where he thinks the polls truly are, rather than simply entering the poll numbers into his model and crunching them.”

Rather than taking a simple average — like RealClearPolitics does — Silver’s model weights polls by his team’s assessment of their quality, and also performs several “adjustments” to account for things like the partisan “lean” of a pollster or the trend lines across different polls.

According to Grim, however, Silver is “just guessing” and his “trend line adjustment” technique is “merely political punditry dressed up as sophisticated mathematical modeling.” Grim also noted that FiveThirtyEight’s model — due to his adjustments — shows Trump more likely than not to win Florida, while the Huffington Post’s calculates her victory there as more likely.

And that, apparently, enraged Silver, whose track record of correctly predicting elections — and explaining how he does it in painstaking, but accessible detail — has made him a celebrity whose very name is synonymous with the art of data-driven prognostication, and whose model is widely considered the gold standard in election forecasting.

Silver complains about other analysts making “assumptions,” but my sense is that Grim and others are doing precisely the opposite: sticking to the numbers. It’s Silver who is making a ton of assumptions about the polls that are unprecedented. And in the end, it’s Silver whose projections are the outlier. He’s got far more at stake in what happens Tuesday, because if this election gets called early (if for example Florida goes comfortably for Clinton), Silver will look stupid, not anyone else. And when someone raises what seem to this amateur to be valid concerns about what Nate Silver is doing, and Silver’s response is to flip out, that doesn’t make me feel very confident in what Silver is saying.

Grim is surely right about this much – we’ll know a lot more early Tuesday night. For now, I trust the numbers. And I think you should too.


One thought on “Nate Silver Loses His Shit At HuffPost’s Ryan Grim

  1. Ben Ross

    Silver is not doing “mere punditry,” but he is making a mathematical mistake. Here’s Sam Wang’s non-technical explanation:
    https://twitter.com/SamWangPhD/status/795083956098101249

    Basically, Silver’s method magnifies recent changes in incoming data and assumes they are going to continue in the future. The less measurement error and back-and-forth fluctuation there is in the data, the better this works. Unfortunately, it’s pretty clear that this year’s polling has lots of both. When that’s the case, Silver’s corrections will make predictions less accurate.

    This phenomenon is known as “overparameterization.” If you want to take a deep dive into this, look at the textbook on time series analysis by Box and Jenkins.

    Reply

Leave a Reply