6 ms·

An accurate model can output a 50-50 prediction. Sure, no problem there. But there is a human bias that does tend to make 50% more likely in these cases. It is

An accurate model can output a 50-50 prediction. Sure, no problem there. But there is a human bias that does tend to make 50% more likely in these cases. It is the maximum percentage you can assign to the uncomfortable possibility without it being higher than the comfortable possibility. 538 systematically magnified this kind of bias when they decided to rate polls, not based on their absolute error, but based on how close their bias was relative to other polls' biases.(https://x.com/andrei__roman/status/1854328028480115144 https://x.com/andrei__roman/status/1854328028480115144) This down-weighted pollsters like Atlas Intel who would've otherwise improved 538's forecast.

culi 2y ago
I'm not sure how to verify your comment since 538 was cut by ABC a month or 2 ago. But Nate Silver's pollster rating methodology is pretty much the same as 538's was during his tenure there and can be found here: https://www.natesilver.net/p/pollster-ratings-silver-bulletin https://www.natesilver.net/p/pollster-ratings-silver-bulleti... It actually explicitly looks for statistical evidence of "herding" (e.g. not publishing poll results that might go against the grain) and penalized those pollsters. In both rating systems, polls that had a long history of going against the grain and being correct, like Ann Seltzer's Iowa poll, were weight very heavily. Seltzer went heavily against the grain 3 elections in a row and was almost completely correct the first 2 times. This year she was off by a massive margin (ultimately costing her her career). Polls that go heavily against the grain but DON'T have a polling history simply aren't weighted heavily in general.
- tantalor 2y ago
  538 was cut by ABC 6 days ago. https://archive.is/E2nre https://archive.is/E2nre
  - culi 2y ago
    Wow thanks. As a former regular reader it's felt a lot longer They've had several major cuts in the past couple of years so maybe that's why it's felt like that
- bitshiftfaced 2y ago
  > I'm not sure how to verify your comment Here's how 538 explains how they factor in bias into their grading: > Think about this another way. If most polls in a race overestimate the Democratic candidate by 10 points in a given election, but Pollster C's surveys overestimate Republicans by 5, there may be something off about the way Pollster C does its polls even if its accuracy is higher. We wouldn't necessarily expect it to keep outperforming other pollsters in subsequent elections since the direction of polling bias bounces around unpredictably from election to election. - https://abcnews.go.com/538/best-pollsters-america/story?id=105563951 https://abcnews.go.com/538/best-pollsters-america/story?id=1...
- mmooss 2y ago
  > Seltzer went heavily against the grain 3 elections in a row and was almost completely correct the first 2 times. This year she was off by a massive margin (ultimately costing her her career). Why did one mistake cost her career?
  - bryanlarsen 2y ago
    Because it pissed off Donald Trump and the media is busy kowtowing to him.
    - culi 2y ago
      you're being downvoted but you're actually pretty much right. She got a lot of backlash from the right accusing her poll of being politically motivated. To the point where they pressured institutions to force her out https://www.nytimes.com/2024/12/19/us/politics/ann-selzer-iowa-trump.html https://www.nytimes.com/2024/12/19/us/politics/ann-selzer-io...
      - bryanlarsen 2y ago
        I'm fine with the downvotes. There are others saying the same thing as me much more cogently -- they should be upweighted and mine downweighted.
      - gowld 2y ago
        Who pressured what institutions? Trump sued her, after she retired as planned. Trump sues everyon.
        mmooss 2y ago
        Trump and others were attacking her and physically threatening her before she retired. See my other comment for evidence.
    - Vaslo 2y ago
      You’re delusional if you think the media kowtows to him. I’m not even going to post sources, a quick search will show even left leaning sites saying the overwhelming reporting is negative about him.
- JumpCrisscross 2y ago
  > since 538 was cut by ABC a month or 2 ago. But Nate Silver's pollster rating methodology is pretty much the same as 538's was during his tenure there Nate took his model with him. After he left, 538 rolled a new model.
  - culi 2y ago
    I'm aware. It's sad to see 538 gone though. I was looking forward to seeing how the new 538 model would compare to silver's model
  - Suppafly 2y ago
    >Nate took his model with him. After he left, 538 rolled a new model. So they let him leave with the only part of it that had value, that's insane. They essentially just paid him for the name?
deleted 2y ago
[deleted]
_ea1k 2y ago
It isn't just human bias that makes this problematic. Their definition of success is that a prediction of 70% should be right about 70/100 times. Give a model a big reward for "success" under this criteria and it will have a tendency to converge on the long term average. As long as the long term average stays the same, no one will be able to say the estimate was wrong. If you've ever seen the YT videos from aspiring data scientists learning sequence prediction, you've inevitably seen a variation from people trying to predict stock prices. "Look how low the loss is!" - Sure, but it gets the direction wrong, and just consistently predicts a close value to the previous day's close. But the model is technically correct, just useless. A 50-50 model for an election might not be giving up and might not be wrong, but it is often useless.
- gowld 2y ago
  50-50 means "I have 0 no useful information". Some facts are more useful than others.
  - rurp 2y ago
    Forecasts will always have some level of uncertainty and if a race is very close a prediction around 50/50 will be the most honest result. A prediction like that does have value, just like a prediction that candidate A is a large favorite, it's informing you about the future state of the world. I'm not sure what you're expecting forecasters to do when they have a 50/50 result. Put a thumb on the scale in a random direction so that you will think they are providing useful information, even though they'll be less accurate?

culi 2y ago

I'm not sure how to verify your comment since 538 was cut by ABC a month or 2 ago. But Nate Silver's pollster rating methodology is pretty much the same as 538's was during his tenure there and can be found here: https://www.natesilver.net/p/pollster-ratings-silver-bulletin https://www.natesilver.net/p/pollster-ratings-silver-bulleti... It actually explicitly looks for statistical evidence of "herding" (e.g. not publishing poll results that might go against the grain) and penalized those pollsters. In both rating systems, polls that had a long history of going against the grain and being correct, like Ann Seltzer's Iowa poll, were weight very heavily. Seltzer went heavily against the grain 3 elections in a row and was almost completely correct the first 2 times. This year she was off by a massive margin (ultimately costing her her career). Polls that go heavily against the grain but DON'T have a polling history simply aren't weighted heavily in general.

tantalor 2y ago
538 was cut by ABC 6 days ago. https://archive.is/E2nre https://archive.is/E2nre
- culi 2y ago
  Wow thanks. As a former regular reader it's felt a lot longer They've had several major cuts in the past couple of years so maybe that's why it's felt like that
bitshiftfaced 2y ago
> I'm not sure how to verify your comment Here's how 538 explains how they factor in bias into their grading: > Think about this another way. If most polls in a race overestimate the Democratic candidate by 10 points in a given election, but Pollster C's surveys overestimate Republicans by 5, there may be something off about the way Pollster C does its polls even if its accuracy is higher. We wouldn't necessarily expect it to keep outperforming other pollsters in subsequent elections since the direction of polling bias bounces around unpredictably from election to election. - https://abcnews.go.com/538/best-pollsters-america/story?id=105563951 https://abcnews.go.com/538/best-pollsters-america/story?id=1...
mmooss 2y ago
> Seltzer went heavily against the grain 3 elections in a row and was almost completely correct the first 2 times. This year she was off by a massive margin (ultimately costing her her career). Why did one mistake cost her career?
- bryanlarsen 2y ago
  Because it pissed off Donald Trump and the media is busy kowtowing to him.
  - culi 2y ago
    you're being downvoted but you're actually pretty much right. She got a lot of backlash from the right accusing her poll of being politically motivated. To the point where they pressured institutions to force her out https://www.nytimes.com/2024/12/19/us/politics/ann-selzer-iowa-trump.html https://www.nytimes.com/2024/12/19/us/politics/ann-selzer-io...
    - bryanlarsen 2y ago
      I'm fine with the downvotes. There are others saying the same thing as me much more cogently -- they should be upweighted and mine downweighted.
    - gowld 2y ago
      Who pressured what institutions? Trump sued her, after she retired as planned. Trump sues everyon.
      - mmooss 2y ago
        Trump and others were attacking her and physically threatening her before she retired. See my other comment for evidence.
  - Vaslo 2y ago
    You’re delusional if you think the media kowtows to him. I’m not even going to post sources, a quick search will show even left leaning sites saying the overwhelming reporting is negative about him.
JumpCrisscross 2y ago
> since 538 was cut by ABC a month or 2 ago. But Nate Silver's pollster rating methodology is pretty much the same as 538's was during his tenure there Nate took his model with him. After he left, 538 rolled a new model.
- culi 2y ago
  I'm aware. It's sad to see 538 gone though. I was looking forward to seeing how the new 538 model would compare to silver's model
- Suppafly 2y ago
  >Nate took his model with him. After he left, 538 rolled a new model. So they let him leave with the only part of it that had value, that's insane. They essentially just paid him for the name?

deleted 2y ago

[deleted]

_ea1k 2y ago

It isn't just human bias that makes this problematic. Their definition of success is that a prediction of 70% should be right about 70/100 times. Give a model a big reward for "success" under this criteria and it will have a tendency to converge on the long term average. As long as the long term average stays the same, no one will be able to say the estimate was wrong. If you've ever seen the YT videos from aspiring data scientists learning sequence prediction, you've inevitably seen a variation from people trying to predict stock prices. "Look how low the loss is!" - Sure, but it gets the direction wrong, and just consistently predicts a close value to the previous day's close. But the model is technically correct, just useless. A 50-50 model for an election might not be giving up and might not be wrong, but it is often useless.

gowld 2y ago
50-50 means "I have 0 no useful information". Some facts are more useful than others.
- rurp 2y ago
  Forecasts will always have some level of uncertainty and if a race is very close a prediction around 50/50 will be the most honest result. A prediction like that does have value, just like a prediction that candidate A is a large favorite, it's informing you about the future state of the world. I'm not sure what you're expecting forecasters to do when they have a 50/50 result. Put a thumb on the scale in a random direction so that you will think they are providing useful information, even though they'll be less accurate?