4 ms·

The article is way too long, and doesn't really talk about Nate Silver much. However, one interesting item is the idea that there is no way to verify these pre

The article is way too long, and doesn't really talk about Nate Silver much. However, one interesting item is the idea that there is no way to verify these predictions. Because they are forecasting one time events, there is no way to validate if their models are in the least bit accurate. If you have a X% chance of event happening and it doesn't, were you wrong? No because it wasn't 100%, and even if you are right, it's not terribly meaningful. It would be interesting to see if a site like five thirty-eight modeled all elections results, and then see how accurate they were across all of their predictions. Maybe you would have to look at all predictions with > 70% chance of happening and see if they got 70% of those correct. However, in the end those predictions seem like they could be done by throwing darts at a dart board and we wouldn't really know.

scarmig 6y ago
It's not as dire as you suggest. To simplify a bit, you can take the set of ~75% predictions created by a model (which can be an actual model, or it can be just the meta model of whatever model Nate Silver wants us to trust right now), and then see what percentage of them were right. If that comes to 75%, then the model is well-calibrated, and it's a solid enough basis to assign a 75% probability to future 75% predictions, which is often as good as you can get when trying to make decisions and plans.
borramakot 6y ago
Boy, have I got the incredibly specific article for that exact question: https://fivethirtyeight.com/features/when-we-say-70-percent-it-really-means-70-percent/ https://fivethirtyeight.com/features/when-we-say-70-percent-...
- borramakot 6y ago
  I don't think this article provides strong evidence that they are well calibrated on the presidential election specifically (sample size N=3), or that they are correctly accounting for rare black swan events, but it does seem to imply that the criticisms about "538 claims victory no matter what because they always have non-zero probabilities" are oversimplified.
  - bryanlarsen 6y ago
    2016 wasn't a black swan event. It was a polling error, which do happen, if rarely. It was not unforseeable, and 538 included a probability of that happening which is why they gave Trump higher chances than most others did. And 538 does do backtesting on elections back to 1972. That's not particularly trustworthy since it invites over-fitting, but internally they do have a little bit more than N=3 to work from.
    - borramakot 6y ago
      (I have fairly minor quibbles with some of Nate's modeling ideas, but I broadly mean to be defending him). I don't mean to imply 2016 was a black swan event- I agree that ~30% was probably as accurate a take as could be achieved (most evidence that seems reasonable to use indicated a lead for Clinton, but that it wouldn't be that surprising for that lead to be overcome). I just mean that the model assumes a fairly normal election environment, without like a huge attack on Election day or something on election day. The N=3 comment was meant specifically for evaluating their calibration, not the data they use for their model.
mabbo 6y ago
> those predictions seem like they could be done by throwing darts at a dart board They are. They're made by throwing 40,000 darts at a dart board. 40,000 simulations of the election based on the polls, historical trends, demographics, and more. The real threat, the real reason why the 2016 election was mostly mis-called, is when the polls themselves make mistakes in a common way. In 2016 it was education that wasn't factored strongly enough. The models can be run on past elections and polling + data before those elections. That's how the develop the models.

scarmig 6y ago

It's not as dire as you suggest. To simplify a bit, you can take the set of ~75% predictions created by a model (which can be an actual model, or it can be just the meta model of whatever model Nate Silver wants us to trust right now), and then see what percentage of them were right. If that comes to 75%, then the model is well-calibrated, and it's a solid enough basis to assign a 75% probability to future 75% predictions, which is often as good as you can get when trying to make decisions and plans.

borramakot 6y ago

Boy, have I got the incredibly specific article for that exact question: https://fivethirtyeight.com/features/when-we-say-70-percent-it-really-means-70-percent/ https://fivethirtyeight.com/features/when-we-say-70-percent-...

borramakot 6y ago
I don't think this article provides strong evidence that they are well calibrated on the presidential election specifically (sample size N=3), or that they are correctly accounting for rare black swan events, but it does seem to imply that the criticisms about "538 claims victory no matter what because they always have non-zero probabilities" are oversimplified.
- bryanlarsen 6y ago
  2016 wasn't a black swan event. It was a polling error, which do happen, if rarely. It was not unforseeable, and 538 included a probability of that happening which is why they gave Trump higher chances than most others did. And 538 does do backtesting on elections back to 1972. That's not particularly trustworthy since it invites over-fitting, but internally they do have a little bit more than N=3 to work from.
  - borramakot 6y ago
    (I have fairly minor quibbles with some of Nate's modeling ideas, but I broadly mean to be defending him). I don't mean to imply 2016 was a black swan event- I agree that ~30% was probably as accurate a take as could be achieved (most evidence that seems reasonable to use indicated a lead for Clinton, but that it wouldn't be that surprising for that lead to be overcome). I just mean that the model assumes a fairly normal election environment, without like a huge attack on Election day or something on election day. The N=3 comment was meant specifically for evaluating their calibration, not the data they use for their model.

mabbo 6y ago

> those predictions seem like they could be done by throwing darts at a dart board They are. They're made by throwing 40,000 darts at a dart board. 40,000 simulations of the election based on the polls, historical trends, demographics, and more. The real threat, the real reason why the 2016 election was mostly mis-called, is when the polls themselves make mistakes in a common way. In 2016 it was education that wasn't factored strongly enough. The models can be run on past elections and polling + data before those elections. That's how the develop the models.