4 ms·

I'd argue that 538's model likely overestimates uncertainty, because it is a benefit to them both ways. Either they spin it that they were perfectly correct in

I'd argue that 538's model likely overestimates uncertainty, because it is a benefit to them both ways. Either they spin it that they were perfectly correct in those 95% of cases where they get it right, or they spin it that they were the least wrong (and therefore the most correct) in the other 5% of cases where everyone gets it wrong.

frgtpsswrdlame 6y ago
It's not a benefit to them both ways, it's only useful to them if there's a substantial amount of those 5%s in which case they're probably right that everyone else is underestimating uncertainty.
ghaff 6y ago
538 has actually written about calibrating past forecasts. https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/
- throwaway2245 6y ago
  Their calibration is good in the context of this article, but there is a reason this article doesn't use their Presidential election model - there isn't enough data to do this calibration here.
chalst 6y ago
At no point did 538 make such a high forecast: their highest forecast was 88%. There's a joke that you should always express 60% confidence in your predictions, since if the prediction pans out, you can claim to be right, but if it fails, you can bring attention to the "two times out of five wrong" part.
- throwaway2245 6y ago
  Yes - thanks for explaining this better.
panarky 6y ago
So you're saying that 538 wanted an uncertain outcome and hacked the model design and parameters to deliver the outcome they wanted?
- throwaway2245 6y ago
  I'm not saying this was intentional, but there is a very high incentive for them to deliver this type of outcome. There is not enough data to back-test their model on a single election which happens every 4 years, so the claim that a 60% prediction is fundamentally very different from a 95% prediction is statistically dubious.
patmorgan23 6y ago
Or maybe predicting the out comes of elections are really hard.

frgtpsswrdlame 6y ago

It's not a benefit to them both ways, it's only useful to them if there's a substantial amount of those 5%s in which case they're probably right that everyone else is underestimating uncertainty.

ghaff 6y ago

538 has actually written about calibrating past forecasts. https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/

throwaway2245 6y ago
Their calibration is good in the context of this article, but there is a reason this article doesn't use their Presidential election model - there isn't enough data to do this calibration here.

chalst 6y ago

At no point did 538 make such a high forecast: their highest forecast was 88%. There's a joke that you should always express 60% confidence in your predictions, since if the prediction pans out, you can claim to be right, but if it fails, you can bring attention to the "two times out of five wrong" part.

throwaway2245 6y ago
Yes - thanks for explaining this better.

panarky 6y ago

So you're saying that 538 wanted an uncertain outcome and hacked the model design and parameters to deliver the outcome they wanted?

throwaway2245 6y ago
I'm not saying this was intentional, but there is a very high incentive for them to deliver this type of outcome. There is not enough data to back-test their model on a single election which happens every 4 years, so the claim that a 60% prediction is fundamentally very different from a 95% prediction is statistically dubious.

patmorgan23 6y ago

Or maybe predicting the out comes of elections are really hard.