6 ms·

My issue with 538’s models is how wide their margins are. I believe they use “results are within the 80% bounds of our model” to gauge whether or not it reasona

by tyre 6y ago

My issue with 538’s models is how wide their margins are. I believe they use “results are within the 80% bounds of our model” to gauge whether or not it reasonably “worked.” But that range is so huge as to be useless. Yes, it only predicts probabilities and can only be as accurate as the underlying data (which seems to be Bad.) But they don’t show/market it as being “anything could happen, this model is wide open and should only be used to rule out crazy outliers”.

dragonwriter 6y ago
> My issue with 538’s models is how wide their margins are. I believe they use “results are within the 80% bounds of our model” to gauge whether or not it reasonably “worked.” They also assess their models in aggregate, but that obviously isn't applicable to the Presidential model in isolation, since it has so few runs; OTOH, it does validate their broad approach while leaving open the possibility that there is something especially wacky with the Presidential model. But without lots of runs of that model, you can't really clearly single that out from chance errors, and if there are isolated instances of unusual effects that show up only in the Presidential election and only, say, for one or two cycles (e.g., if there is an effect around Trump which is atypical but not new normal) that aren't accounted for but also aren't long-term features, you probably can't even reliably detect them once there are large numbers of runs of the Presidential model. Unfortunately, validating a statistical predictive model isn't itself a a probabilistic exercise subject to small-N problems.
divbzero 6y ago
I don’t see how wide margins are a fault. To me that’s just being honest about the uncertainty. I also don’t think they hide this uncertainty. The model results I’ve seen on FiveThirtyEight always feature a distribution that shows the electoral margin could end up being anything in a wide –X to +Y range.
- jonstewart 6y ago
  The problem’s that 538 is selling certainty. That’s where their pixels go, with the implication that they’re big math nerds who can divine the future based on combining lots of polls and other data sources. If it all amounts to “eh, we don’t really know,” then 538 doesn’t have much of a business model.
- SilasX 6y ago
  >I don’t see how wide margins are a fault. To me that’s just being honest about the uncertainty. Something still seems off about that. How do you do that many large polls of that many primaries and have such error still? Sure, there's going to be sampling error, hitting unrepresentative pockets and so on, but ... why can't they get any better after all this time (i.e. there was a previous Trump election) and research on the demographics?
sdenton4 6y ago
Well, we can probably agree that if the margins were narrower, they would be more flatly wrong. So sounds like a feature, to me. If you want to maximize the probability of the correct outcome in the face of lots of uncertainty, you get wide distributions.
bryanlarsen 6y ago
https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/ In my opinion 538 nailed it in both 2016 and 2020. In 2016 they said Trump needed a full standard polling error to win. In 2020 they said he would need more than a full standard polling error. The polls were wrong, and 538 told us what would happen in that scenario.
- rayiner 6y ago
  Right. The problem is the polling.
  - thu2111 6y ago
    I've read that US polls are much smaller than equivalent UK polls, like sample sizes 90% lower, sometimes just a few hundred people whereas political polls in the UK use more like a thousand or two. It seems like an odd discrepancy. But really. If the polls are so unreliable then at some point people like FiveThirtyEight or the Economist need to understand that and stop incorporating them into models (which may mean not attempting to forecast at all). Just pointing the finger of blame around only gets you so far.
    - joaomacp 6y ago
      I also tend to agree with Peter Thiel's point on The Portal podcast: a significant number of Americans lie on polls because they're ashamed of voting for Donald Trump. My opinion: I think deep down these people feel they are rationally wrong for voting in a president that promised to, and now is actually attacking USA's democracy and freedom which used to be touted as America's strong point. They know they're acting on instinct, and people tend to hide their instincts in public.
      - skinkestek 6y ago
        Have my upvote. > My opinion: I think deep down these people feel they are rationally wrong for voting in a president that promised to, and now is actually attacking USA's democracy and freedom which used to be touted as America's strong point. I think you underestimate peoples distrust of mainstream politicians. Many find Trumps suggestions abhorrent, but still support investigations. Think if this happened to "your" candidate: - There's no doubt that Twitter and Facebook has been uncritically spreading false rumors about Trump only to turn right around and supress the H.Biden leaks, even after it was clear at least some of the juicy mails was cryptographically verified. If Twitter had been posting a warning everytime someone pretended the "fine folks on both sides" hoax then they would have been in a better position now. Same if they had come up with the rule against hacked materials when Trumps tax papers supposedly leaked. - Ballot collections - Sweeping changes to election laws etc etc. I dislike Trump, but it is hard to not get some sympathy for someone who has managed to get a number of really fantastic peace deals, have finally started to improve lives of people in the midwest - without hurting the coasts - and yet is hounded by almost every mainstream media outlet.

by tyre 6y ago

dragonwriter 6y ago

> My issue with 538’s models is how wide their margins are. I believe they use “results are within the 80% bounds of our model” to gauge whether or not it reasonably “worked.” They also assess their models in aggregate, but that obviously isn't applicable to the Presidential model in isolation, since it has so few runs; OTOH, it does validate their broad approach while leaving open the possibility that there is something especially wacky with the Presidential model. But without lots of runs of that model, you can't really clearly single that out from chance errors, and if there are isolated instances of unusual effects that show up only in the Presidential election and only, say, for one or two cycles (e.g., if there is an effect around Trump which is atypical but not new normal) that aren't accounted for but also aren't long-term features, you probably can't even reliably detect them once there are large numbers of runs of the Presidential model. Unfortunately, validating a statistical predictive model isn't itself a a probabilistic exercise subject to small-N problems.

divbzero 6y ago

I don’t see how wide margins are a fault. To me that’s just being honest about the uncertainty. I also don’t think they hide this uncertainty. The model results I’ve seen on FiveThirtyEight always feature a distribution that shows the electoral margin could end up being anything in a wide –X to +Y range.

jonstewart 6y ago
The problem’s that 538 is selling certainty. That’s where their pixels go, with the implication that they’re big math nerds who can divine the future based on combining lots of polls and other data sources. If it all amounts to “eh, we don’t really know,” then 538 doesn’t have much of a business model.
SilasX 6y ago
>I don’t see how wide margins are a fault. To me that’s just being honest about the uncertainty. Something still seems off about that. How do you do that many large polls of that many primaries and have such error still? Sure, there's going to be sampling error, hitting unrepresentative pockets and so on, but ... why can't they get any better after all this time (i.e. there was a previous Trump election) and research on the demographics?

sdenton4 6y ago

Well, we can probably agree that if the margins were narrower, they would be more flatly wrong. So sounds like a feature, to me. If you want to maximize the probability of the correct outcome in the face of lots of uncertainty, you get wide distributions.

bryanlarsen 6y ago

https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/ In my opinion 538 nailed it in both 2016 and 2020. In 2016 they said Trump needed a full standard polling error to win. In 2020 they said he would need more than a full standard polling error. The polls were wrong, and 538 told us what would happen in that scenario.

rayiner 6y ago
Right. The problem is the polling.
- thu2111 6y ago
  I've read that US polls are much smaller than equivalent UK polls, like sample sizes 90% lower, sometimes just a few hundred people whereas political polls in the UK use more like a thousand or two. It seems like an odd discrepancy. But really. If the polls are so unreliable then at some point people like FiveThirtyEight or the Economist need to understand that and stop incorporating them into models (which may mean not attempting to forecast at all). Just pointing the finger of blame around only gets you so far.
  - joaomacp 6y ago
    I also tend to agree with Peter Thiel's point on The Portal podcast: a significant number of Americans lie on polls because they're ashamed of voting for Donald Trump. My opinion: I think deep down these people feel they are rationally wrong for voting in a president that promised to, and now is actually attacking USA's democracy and freedom which used to be touted as America's strong point. They know they're acting on instinct, and people tend to hide their instincts in public.
    - skinkestek 6y ago
      Have my upvote. > My opinion: I think deep down these people feel they are rationally wrong for voting in a president that promised to, and now is actually attacking USA's democracy and freedom which used to be touted as America's strong point. I think you underestimate peoples distrust of mainstream politicians. Many find Trumps suggestions abhorrent, but still support investigations. Think if this happened to "your" candidate: - There's no doubt that Twitter and Facebook has been uncritically spreading false rumors about Trump only to turn right around and supress the H.Biden leaks, even after it was clear at least some of the juicy mails was cryptographically verified. If Twitter had been posting a warning everytime someone pretended the "fine folks on both sides" hoax then they would have been in a better position now. Same if they had come up with the rule against hacked materials when Trumps tax papers supposedly leaked. - Ballot collections - Sweeping changes to election laws etc etc. I dislike Trump, but it is hard to not get some sympathy for someone who has managed to get a number of really fantastic peace deals, have finally started to improve lives of people in the midwest - without hurting the coasts - and yet is hounded by almost every mainstream media outlet.