8 ms·

How is being slightly less wrong than everyone else "having a great track record"? Serious question. Because I find it hard to take any of them seriously after

How is being slightly less wrong than everyone else "having a great track record"? Serious question. Because I find it hard to take any of them seriously after the debacle that was 2016.

abnry 6y ago
They said Trump had a 1 in 4 chance. That's very high. NYT had something like 1 in 20 chance for Trump.
- ceilingcorner 6y ago
  That doesn't really answer my question. It only indicates that they were slightly less wrong than every other media source, not that they have a good model. If I had a laptop that only worked 1/4th of the time, rather than 1/20th of the time, would that make it a reliable laptop? I don't think so.
  - delecti 6y ago
    It's not "wrong" to predict a low chance for something that eventually happens. Unlikely events can happen.
    - ceilingcorner 6y ago
      That isn't the criticism. The criticism is the appellation of it being "unlikely." For example: anyone paying attention to the Rust Belt ±1980-2016 would have dramatically upped Trump's chances in Pennsylvania and Michigan. FiveThirtyEight had Hillary with 70%+ chance of winning both, which to me, shows a deep ignorance of actual cultural factors.
      - yongjik 6y ago
        > anyone paying attention to the Rust Belt ±1980-2016 would have dramatically upped Trump's chances in Pennsylvania and Michigan. There was a very decent chance that Clinton could have won in 2016 (if any factor had gone slightly better for her), and if that had happened, nobody would be saying this now. This is literal hindsight bias.
        ceilingcorner 6y ago
        Aren't you doing the exact same thing that you're accusing me of? My view is simple: the media completely, totally got 2016 wrong, mostly for sociological reasons. The people making the predictions simply had a huge blind spot. Brexit is another similar situation. The fact that Hillary almost won or Brexit almost didn't happen isn't really the point, because both things were never expected to be even remotely that close. Had the predictions been "Pennsylvania will be close", it would be relevant, but those weren't the predictions.
  - pedrosorio 6y ago
    If they were wrong, but “less wrong” than all others, you should pick their model (unless you have an oracle, because the alternative - flipping a coin or “relying on your intuition” is rarely better). Also, it doesnt make sense to look at a single prediction to evaluate a model. Out of all the predictions they have made (did you look at individual state predictions?), how many were correct (and how confident were they?) - how many were wrong (and how close to 50% were they?). That is how you evaluate a model (aka cross entropy)
  - Tyr42 6y ago
    If I tell you you're not likely to get two heads in a row, and you do, does that make me un-reliable? It's unfortunate we can't just run the election again a few times, and actually find the rate at which Trump is elected given the polls. And it's not empty signalling if 538 assigned Trump a higher chance of winning; they were pretty much the only ones saying he has a chance. That is why people think the models are useful.
  - tracer4201 6y ago
    It seems like you are conflating probabilities with absolute certainties. If I had a 1:10 probability of winning the lottery, I would probably take it. If I had a 1:20 probability of getting injured if I leave my house today, I’d stay home. If I did get out but didn’t get injured doesn’t mean the model was wrong.
  - shajznnckfke 6y ago
    If you have a die that rolls a one 1/6 of the time, do you consider the die wrong?
    - ceilingcorner 6y ago
      No, but I don't consider it useful toward predicting the outcome, which is of course what this is all about.
      - shajznnckfke 6y ago
        If there was sufficient data to assign a 0% or 100% probability to an event, that’s what a forecaster should do. If there isn’t sufficient data, then anyone who claims there is a sure thing is a charlatan.
      - thomasmeeks 6y ago
        Say more about "but I don't consider it useful toward predicting the outcome" How else would one predict the outcome of a die roll, specifically?
  - abnry 6y ago
    If everyone was wrong, it is reasonable to believe a low probability event occurred. However, given the extent to which people predicted a Trump loss (say 1 in 20), which is significantly rarer, given that the event occurred it suggests the model that predicted a Trump win with the greatest probability to likely be a more accurate model.
gamblor956 6y ago
They were generally correct with their prediction except in 3 states where nobody had been doing detailed polling because the pollsters didn't think it would matter. It simply wouldn't have been possible for the models to be more accurate with the data they had. As they say, bad data in, bad data out. That's why this time around the pollsters made sure to be more thorough in their polling.
- ceilingcorner 6y ago
  > That's why this time around the pollsters made sure to be more thorough in their polling. "This time is different." I've heard that enough times to be highly skeptical. I'm also deeply skeptical of the notion that polling is even remotely correlated to actual results. Cultural and historical trends play a drastically higher role and are almost always left out.
  - chki 6y ago
    > I'm also deeply skeptical of the notion that polling is even remotely correlated to actual results. But it has been strongly correlated to the results in basically all elections so far in all democracies on the planet. Taking 2016 as an example there has been a very strong correlation between polling and the results. The national polling averages were only 3 points off from the actual result. If that's not correlated I don't know what is.
    - ceilingcorner 6y ago
      Polarization and the unacceptability of publicly saying "I voted for X" also didn't really exist prior to 2016. The fear of getting doxxed, combined with a record low level of trust in institutions and the media, leads to skepticism toward answering polls truthfully, IMO.
      - gamblor956 6y ago
        While polarization is bad now, it's nowhere close to historical extremes, and it's not even as bad as it was during the post-Vietnam era only a few decades ago. As for fear of doxing: plenty of people openly supported and voted for George Wallace (a noted white Supremacist) back in the day, and even Roy Moore (accused pedophile) just 2 years ago. Proud Boys members openly pose for the cameras even as they espouse racist views, and QAnon members brag about being part of QAnon. The purported shy Trump voter? Doesn't exist. https://fivethirtyeight.com/features/trump-supporters-arent-shy-but-polls-could-still-be-missing-some-of-them/ https://fivethirtyeight.com/features/trump-supporters-arent-...
massysett 6y ago
Saying that the ultimate outcome had a 1 in 4 chance is not wrong, slightly wrong, or less wrong. If the weatherman says there's a 1 in 4 chance of rain, and it rains, he wasn't wrong.
- ceilingcorner 6y ago
  No, but it means that the weatherman isn't particularly effective at forecasting the weather.
  - noirbot 6y ago
    Only if they say that every day, and it rains every day. If the most likely outcome happened every time, then the model is likely wrong/under-confident. The prediction is never going to be 100% accurate until the event is happening/has happened. Up until then, there's always a chance you're wrong or something can change. Being wrong once isn't necessarily a sign that the whole system is messed up.
  - bart_spoon 6y ago
    No it doesn’t, and this is a fundamental misunderstanding of how probabilistic forecasting works. If it rains 9 out of 10 times a weatherman says there is a 30% chance of rain, they are a bad weatherman, but they aren’t much worse than if it rained 0 out of 10 times they predicted a 30% chance of rain. A weatherman accurately assessing the probability of the weather forecast would see it rain around 3 out of 10 days they say there is a 30% chance of rain.
- swsieber 6y ago
  To take it a step further, if that was the forecast 4 days in a row, you would expect it to rain one of those days.
  - warrenq 6y ago
    Not necessarily. There is a 0.750.750.75*0.75 or a 31% chance of no rain at all
khuey 6y ago
This is a fundamental misunderstanding of probability. Low probability events do happen, and it doesn't inherently mean the estimated probability was wrong.
- ceilingcorner 6y ago
  I am criticizing their labeling of Trump's win as low probability.
  - khuey 6y ago
    "He won" is not sufficient evidence upon which to do that.
    - ceilingcorner 6y ago
      Nowhere did I say it is. I simply think that, if one were following the right information, his win was not as unexpected as the coastal media presented it as being. Personally I would have put it about 60-40 Hillary-Trump.
      - wtallis 6y ago
        You started off by calling the 2016 prediction a debacle, but now you're saying you would have put the odds about 12 percentage points differently. That doesn't seem like a big enough disagreement to warrant the kind of vehement criticism you're throwing around.
        ceilingcorner 6y ago
        I called it a debacle because 99.9% of media sources, pundits, politicians, political figures, or anyone else thought that Hillary had anything less than a guaranteed win. I'm simply suggesting it was actually always a close race, but that the media ignored this because it went against their ideological model / they weren't familiar with places like the Rust Belt.
pjanoman 6y ago
At the end of the 2016 election, Donald Trump had a predicted 1/4 chance of winning the presidency. Does this seem like a massive debacle that Trump won under these conditions? Not to me. https://projects.fivethirtyeight.com/2016-election-forecast/ https://projects.fivethirtyeight.com/2016-election-forecast/

How is being slightly less wrong than everyone else "having a great track record"? Serious question. Because I find it hard to take any of them seriously after the debacle that was 2016.

abnry 6y ago

They said Trump had a 1 in 4 chance. That's very high. NYT had something like 1 in 20 chance for Trump.

ceilingcorner 6y ago
That doesn't really answer my question. It only indicates that they were slightly less wrong than every other media source, not that they have a good model. If I had a laptop that only worked 1/4th of the time, rather than 1/20th of the time, would that make it a reliable laptop? I don't think so.
- delecti 6y ago
  It's not "wrong" to predict a low chance for something that eventually happens. Unlikely events can happen.
  - ceilingcorner 6y ago
    That isn't the criticism. The criticism is the appellation of it being "unlikely." For example: anyone paying attention to the Rust Belt ±1980-2016 would have dramatically upped Trump's chances in Pennsylvania and Michigan. FiveThirtyEight had Hillary with 70%+ chance of winning both, which to me, shows a deep ignorance of actual cultural factors.
    - yongjik 6y ago
      > anyone paying attention to the Rust Belt ±1980-2016 would have dramatically upped Trump's chances in Pennsylvania and Michigan. There was a very decent chance that Clinton could have won in 2016 (if any factor had gone slightly better for her), and if that had happened, nobody would be saying this now. This is literal hindsight bias.
      - ceilingcorner 6y ago
        Aren't you doing the exact same thing that you're accusing me of? My view is simple: the media completely, totally got 2016 wrong, mostly for sociological reasons. The people making the predictions simply had a huge blind spot. Brexit is another similar situation. The fact that Hillary almost won or Brexit almost didn't happen isn't really the point, because both things were never expected to be even remotely that close. Had the predictions been "Pennsylvania will be close", it would be relevant, but those weren't the predictions.
- pedrosorio 6y ago
  If they were wrong, but “less wrong” than all others, you should pick their model (unless you have an oracle, because the alternative - flipping a coin or “relying on your intuition” is rarely better). Also, it doesnt make sense to look at a single prediction to evaluate a model. Out of all the predictions they have made (did you look at individual state predictions?), how many were correct (and how confident were they?) - how many were wrong (and how close to 50% were they?). That is how you evaluate a model (aka cross entropy)
- Tyr42 6y ago
  If I tell you you're not likely to get two heads in a row, and you do, does that make me un-reliable? It's unfortunate we can't just run the election again a few times, and actually find the rate at which Trump is elected given the polls. And it's not empty signalling if 538 assigned Trump a higher chance of winning; they were pretty much the only ones saying he has a chance. That is why people think the models are useful.
- tracer4201 6y ago
  It seems like you are conflating probabilities with absolute certainties. If I had a 1:10 probability of winning the lottery, I would probably take it. If I had a 1:20 probability of getting injured if I leave my house today, I’d stay home. If I did get out but didn’t get injured doesn’t mean the model was wrong.
- shajznnckfke 6y ago
  If you have a die that rolls a one 1/6 of the time, do you consider the die wrong?
  - ceilingcorner 6y ago
    No, but I don't consider it useful toward predicting the outcome, which is of course what this is all about.
    - shajznnckfke 6y ago
      If there was sufficient data to assign a 0% or 100% probability to an event, that’s what a forecaster should do. If there isn’t sufficient data, then anyone who claims there is a sure thing is a charlatan.
    - thomasmeeks 6y ago
      Say more about "but I don't consider it useful toward predicting the outcome" How else would one predict the outcome of a die roll, specifically?
- abnry 6y ago
  If everyone was wrong, it is reasonable to believe a low probability event occurred. However, given the extent to which people predicted a Trump loss (say 1 in 20), which is significantly rarer, given that the event occurred it suggests the model that predicted a Trump win with the greatest probability to likely be a more accurate model.

gamblor956 6y ago

They were generally correct with their prediction except in 3 states where nobody had been doing detailed polling because the pollsters didn't think it would matter. It simply wouldn't have been possible for the models to be more accurate with the data they had. As they say, bad data in, bad data out. That's why this time around the pollsters made sure to be more thorough in their polling.

ceilingcorner 6y ago
> That's why this time around the pollsters made sure to be more thorough in their polling. "This time is different." I've heard that enough times to be highly skeptical. I'm also deeply skeptical of the notion that polling is even remotely correlated to actual results. Cultural and historical trends play a drastically higher role and are almost always left out.
- chki 6y ago
  > I'm also deeply skeptical of the notion that polling is even remotely correlated to actual results. But it has been strongly correlated to the results in basically all elections so far in all democracies on the planet. Taking 2016 as an example there has been a very strong correlation between polling and the results. The national polling averages were only 3 points off from the actual result. If that's not correlated I don't know what is.
  - ceilingcorner 6y ago
    Polarization and the unacceptability of publicly saying "I voted for X" also didn't really exist prior to 2016. The fear of getting doxxed, combined with a record low level of trust in institutions and the media, leads to skepticism toward answering polls truthfully, IMO.
    - gamblor956 6y ago
      While polarization is bad now, it's nowhere close to historical extremes, and it's not even as bad as it was during the post-Vietnam era only a few decades ago. As for fear of doxing: plenty of people openly supported and voted for George Wallace (a noted white Supremacist) back in the day, and even Roy Moore (accused pedophile) just 2 years ago. Proud Boys members openly pose for the cameras even as they espouse racist views, and QAnon members brag about being part of QAnon. The purported shy Trump voter? Doesn't exist. https://fivethirtyeight.com/features/trump-supporters-arent-shy-but-polls-could-still-be-missing-some-of-them/ https://fivethirtyeight.com/features/trump-supporters-arent-...

massysett 6y ago

Saying that the ultimate outcome had a 1 in 4 chance is not wrong, slightly wrong, or less wrong. If the weatherman says there's a 1 in 4 chance of rain, and it rains, he wasn't wrong.

ceilingcorner 6y ago
No, but it means that the weatherman isn't particularly effective at forecasting the weather.
- noirbot 6y ago
  Only if they say that every day, and it rains every day. If the most likely outcome happened every time, then the model is likely wrong/under-confident. The prediction is never going to be 100% accurate until the event is happening/has happened. Up until then, there's always a chance you're wrong or something can change. Being wrong once isn't necessarily a sign that the whole system is messed up.
- bart_spoon 6y ago
  No it doesn’t, and this is a fundamental misunderstanding of how probabilistic forecasting works. If it rains 9 out of 10 times a weatherman says there is a 30% chance of rain, they are a bad weatherman, but they aren’t much worse than if it rained 0 out of 10 times they predicted a 30% chance of rain. A weatherman accurately assessing the probability of the weather forecast would see it rain around 3 out of 10 days they say there is a 30% chance of rain.
swsieber 6y ago
To take it a step further, if that was the forecast 4 days in a row, you would expect it to rain one of those days.
- warrenq 6y ago
  Not necessarily. There is a 0.750.750.75*0.75 or a 31% chance of no rain at all

khuey 6y ago

This is a fundamental misunderstanding of probability. Low probability events do happen, and it doesn't inherently mean the estimated probability was wrong.

ceilingcorner 6y ago
I am criticizing their labeling of Trump's win as low probability.
- khuey 6y ago
  "He won" is not sufficient evidence upon which to do that.
  - ceilingcorner 6y ago
    Nowhere did I say it is. I simply think that, if one were following the right information, his win was not as unexpected as the coastal media presented it as being. Personally I would have put it about 60-40 Hillary-Trump.
    - wtallis 6y ago
      You started off by calling the 2016 prediction a debacle, but now you're saying you would have put the odds about 12 percentage points differently. That doesn't seem like a big enough disagreement to warrant the kind of vehement criticism you're throwing around.
      - ceilingcorner 6y ago
        I called it a debacle because 99.9% of media sources, pundits, politicians, political figures, or anyone else thought that Hillary had anything less than a guaranteed win. I'm simply suggesting it was actually always a close race, but that the media ignored this because it went against their ideological model / they weren't familiar with places like the Rust Belt.

pjanoman 6y ago

At the end of the 2016 election, Donald Trump had a predicted 1/4 chance of winning the presidency. Does this seem like a massive debacle that Trump won under these conditions? Not to me. https://projects.fivethirtyeight.com/2016-election-forecast/ https://projects.fivethirtyeight.com/2016-election-forecast/