5 ms·

I had complaints about 538, especially the early days, but don't understand this critique at all. A 30% chance hitting is completely unremarkable, and it was a

by rurp 5mo ago

I had complaints about 538, especially the early days, but don't understand this critique at all. A 30% chance hitting is completely unremarkable, and it was a perfectly reasonable reading of the evidence at the time. Nate isn't wrong that conventional wisdom was way off, with even supposedly statistical models giving Hillary a 99% chance of winning. Elections, like many things, have some inherent uncertainty. A several point polling error is normal, so a candidate who is down a couple points on election day has a decent shot of winning.

Retric 5mo ago
Not just that, predictions also impact voter participation.
munchler 5mo ago
I think this is all true, but it dodges the bigger issue. A presidential election has a binary outcome: yes/no, win/lose. If your statistical model doesn’t contain this single bit in its output, then it doesn’t meet the minimum requirement for being a prediction. Now you might say that it was on me as a consumer to understand this in 2016, but I remember the look of total shock on Nate Silver’s face when he called the winner on live TV that night, so clearly he didn’t really understand it either. Lesson learned for all of us, I guess.
- causalmodels 5mo ago
  > If your statistical model doesn’t contain this single bit in its output, then it doesn’t meet the minimum requirement for being a prediction. I would be fascinated to know how you model a coin flip.
  - munchler 5mo ago
    Well, that’s my point. You can model a coin flip, but you can’t predict it. A model and a prediction are two different things.
lacewing 5mo ago
I don't understand why this is surprising. People didn't go to FiveThirtyEight to marvel the science behind it. The science was just supposed to give you what you came there for: the actual election results. In the end, it turned out that predicting elections is still very hard, and that for all the fanfare, FiveThirtyEight performed only slightly better than what you could find in any other reputable newspaper, so it kinda lost its appeal.
- hungryhobbit 5mo ago
  538 was never about magically making polls more reliable, and only people that don't understand what polls are could think that (caveat: lots of people don't understand how polls work). 538 was about analyzing and communicating the information from those polls in an easily accessible form. If you came to the site for that, you weren't mad that they "predicted poorly something that was impossible to predict from the data sources they used" ... you were just mad at Trump for winning (despite polls suggesting otherwise).
  - lacewing 5mo ago
    Again, I don't think any of this matters. People were not coming there to have "information communicated to them". They were coming there for the satisfaction of knowing the results before everyone else. And FiveThirtyEight couldn't realistically deliver on that.
    - nomel 5mo ago
      I'm very curious to see how polymarket fairs, compared to the news agencies. I suspect prediction markets will be the norm, going forward. Polls can't fully capture the element of anonymity that's required for an accurate poll of something controversial.
      - bombcar 5mo ago
        My experience was that prediction markets were lagging indicators and basically followed something akin to an aggregate opinion of polls. This is especially viewable if you watch them during the 2020 election.
ngriffiths 5mo ago
Discussion of stats models is always complicated by the fact that a lot of people will read "30%" as a "no" prediction and claim your model is wrong if the thing happens. On the one hand, one strategy is to "hide" the numbers a bit behind a blaring headline that says "we are not sure!!" It's a bit of an art to decide when to be "sure" or not. On the other hand, in research for example you can just say screw it, I care if the correct people are correct, not if a bunch of wrong people are wrong. I feel like the correct strategy for 538 when it was actually niche was to be precise, but then it went viral and maybe should've hit the IDK button much harder and more often after that.
- gh02t 5mo ago
  The real caveat is that 538 was a Monte Carlo model, and is only as good as its inputs. "Here's what the current spread in polling numbers is *given our model and the current polling and their reported uncertainties.*" Polling uncertainties are themselves computed under certain models, and those models are subject to errors. I don't think 538 hid this, but it's a difficult caveat for people to reason about because the sorts of modeling errors that have the most influence usually represent "unknown unknowns".
  - bryanlarsen 5mo ago
    Regularly referring to that ~30% spread as "one polling error" made this a lot more understandable than most statistics for many people.
  - this_user 5mo ago
    Building a model for predicting the ultimate winner of a US presidential election is particularly difficult, because you are dealing with noisy input data and nonlinear effects, i.e. just a few thousand votes in a few key states can completely flip the outcome. If you then have poorly calibrated polls with a large margin of error, there is really nothing much you can do. On the other hand, it does raise the question how valuable the 538 models for something like this really are if the outcome is a coin flip anyway.
deleted 5mo ago
[deleted]
Sparkle-san 5mo ago
I can't find the source anymore since 538 is no more, and I recall Nate even describing what could (and did) happen, which was that one swing state moving to the right had a high likelihood of them all moving to the right.
- rurp 5mo ago
  Yeah, Nate has talked a number of times about polling errors being correlated across states. In fact that's probably one of the most common mistakes models can make, treating correlated inputs as independent. There's a long history of that mistake in financial markets as well. In 2024 the single most likely outcome his model had was trump winning all 7 swing states. The second most likely was Harris winning all 7.
FergusArgyll 5mo ago
It was 30% in the end, before the nomination it famously gave him a 2% chance of getting nominated. All the talk about 30% is disingenuous https://en.wikipedia.org/wiki/FiveThirtyEight#2016_U.S._elections https://en.wikipedia.org/wiki/FiveThirtyEight#2016_U.S._elec...
- lmm 5mo ago
  They had him at 30% at a time when most reputable media had him in single digits at best. You might not remember now, but the result of that election was a real shock.
- Brendinooo 5mo ago
  From the article: > Being dismissive of Trump’s chances early on in the Republican primary was the biggest analytical mistake of my career, and I think I deserved a lot of blame for that. He went on to write a whole retrospective, which is exactly what you'd want to see: https://web.archive.org/web/20160527093831/https://fivethirtyeight.com/features/how-i-acted-like-a-pundit-and-screwed-up-on-donald-trump/ https://web.archive.org/web/20160527093831/https://fivethirt... So unless your standard is "pollsters must be perfect at all times", I dunno what else should be asked of Silver here.
  - FergusArgyll 5mo ago
    I'm fine with him[0]. I don't like the apologists who explain patronizingly that 30% odds happen 3/10 times. He gave Trump much much lower odds early on and only put it at 30 in the last few weeks. [0] I'd say the better defense of him is: the whole model is built on polls, the pollsters stink, people should complain about pollsters, not their aggregators.

by rurp 5mo ago

Retric 5mo ago

Not just that, predictions also impact voter participation.

munchler 5mo ago

I think this is all true, but it dodges the bigger issue. A presidential election has a binary outcome: yes/no, win/lose. If your statistical model doesn’t contain this single bit in its output, then it doesn’t meet the minimum requirement for being a prediction. Now you might say that it was on me as a consumer to understand this in 2016, but I remember the look of total shock on Nate Silver’s face when he called the winner on live TV that night, so clearly he didn’t really understand it either. Lesson learned for all of us, I guess.

causalmodels 5mo ago
> If your statistical model doesn’t contain this single bit in its output, then it doesn’t meet the minimum requirement for being a prediction. I would be fascinated to know how you model a coin flip.
- munchler 5mo ago
  Well, that’s my point. You can model a coin flip, but you can’t predict it. A model and a prediction are two different things.

lacewing 5mo ago

I don't understand why this is surprising. People didn't go to FiveThirtyEight to marvel the science behind it. The science was just supposed to give you what you came there for: the actual election results. In the end, it turned out that predicting elections is still very hard, and that for all the fanfare, FiveThirtyEight performed only slightly better than what you could find in any other reputable newspaper, so it kinda lost its appeal.

hungryhobbit 5mo ago
538 was never about magically making polls more reliable, and only people that don't understand what polls are could think that (caveat: lots of people don't understand how polls work). 538 was about analyzing and communicating the information from those polls in an easily accessible form. If you came to the site for that, you weren't mad that they "predicted poorly something that was impossible to predict from the data sources they used" ... you were just mad at Trump for winning (despite polls suggesting otherwise).
- lacewing 5mo ago
  Again, I don't think any of this matters. People were not coming there to have "information communicated to them". They were coming there for the satisfaction of knowing the results before everyone else. And FiveThirtyEight couldn't realistically deliver on that.
  - nomel 5mo ago
    I'm very curious to see how polymarket fairs, compared to the news agencies. I suspect prediction markets will be the norm, going forward. Polls can't fully capture the element of anonymity that's required for an accurate poll of something controversial.
    - bombcar 5mo ago
      My experience was that prediction markets were lagging indicators and basically followed something akin to an aggregate opinion of polls. This is especially viewable if you watch them during the 2020 election.

ngriffiths 5mo ago

Discussion of stats models is always complicated by the fact that a lot of people will read "30%" as a "no" prediction and claim your model is wrong if the thing happens. On the one hand, one strategy is to "hide" the numbers a bit behind a blaring headline that says "we are not sure!!" It's a bit of an art to decide when to be "sure" or not. On the other hand, in research for example you can just say screw it, I care if the correct people are correct, not if a bunch of wrong people are wrong. I feel like the correct strategy for 538 when it was actually niche was to be precise, but then it went viral and maybe should've hit the IDK button much harder and more often after that.

gh02t 5mo ago
The real caveat is that 538 was a Monte Carlo model, and is only as good as its inputs. "Here's what the current spread in polling numbers is *given our model and the current polling and their reported uncertainties.*" Polling uncertainties are themselves computed under certain models, and those models are subject to errors. I don't think 538 hid this, but it's a difficult caveat for people to reason about because the sorts of modeling errors that have the most influence usually represent "unknown unknowns".
- bryanlarsen 5mo ago
  Regularly referring to that ~30% spread as "one polling error" made this a lot more understandable than most statistics for many people.
- this_user 5mo ago
  Building a model for predicting the ultimate winner of a US presidential election is particularly difficult, because you are dealing with noisy input data and nonlinear effects, i.e. just a few thousand votes in a few key states can completely flip the outcome. If you then have poorly calibrated polls with a large margin of error, there is really nothing much you can do. On the other hand, it does raise the question how valuable the 538 models for something like this really are if the outcome is a coin flip anyway.

deleted 5mo ago

[deleted]

Sparkle-san 5mo ago

I can't find the source anymore since 538 is no more, and I recall Nate even describing what could (and did) happen, which was that one swing state moving to the right had a high likelihood of them all moving to the right.

rurp 5mo ago
Yeah, Nate has talked a number of times about polling errors being correlated across states. In fact that's probably one of the most common mistakes models can make, treating correlated inputs as independent. There's a long history of that mistake in financial markets as well. In 2024 the single most likely outcome his model had was trump winning all 7 swing states. The second most likely was Harris winning all 7.

FergusArgyll 5mo ago

It was 30% in the end, before the nomination it famously gave him a 2% chance of getting nominated. All the talk about 30% is disingenuous https://en.wikipedia.org/wiki/FiveThirtyEight#2016_U.S._elections https://en.wikipedia.org/wiki/FiveThirtyEight#2016_U.S._elec...

lmm 5mo ago
They had him at 30% at a time when most reputable media had him in single digits at best. You might not remember now, but the result of that election was a real shock.
Brendinooo 5mo ago
From the article: > Being dismissive of Trump’s chances early on in the Republican primary was the biggest analytical mistake of my career, and I think I deserved a lot of blame for that. He went on to write a whole retrospective, which is exactly what you'd want to see: https://web.archive.org/web/20160527093831/https://fivethirtyeight.com/features/how-i-acted-like-a-pundit-and-screwed-up-on-donald-trump/ https://web.archive.org/web/20160527093831/https://fivethirt... So unless your standard is "pollsters must be perfect at all times", I dunno what else should be asked of Silver here.
- FergusArgyll 5mo ago
  I'm fine with him[0]. I don't like the apologists who explain patronizingly that 30% odds happen 3/10 times. He gave Trump much much lower odds early on and only put it at 30 in the last few weeks. [0] I'd say the better defense of him is: the whole model is built on polls, the pollsters stink, people should complain about pollsters, not their aggregators.