6 ms·

Some of the other models for the last election were really bad. Giving something like a 95-98% chance to Hillary was arguably a fundamental failure. I found it

Some of the other models for the last election were really bad. Giving something like a 95-98% chance to Hillary was arguably a fundamental failure. I found it very odd how they arrived at those numbers by e.g. treating every state as a separate chance while mostly ignoring that those results are not uncorrelated. I think he does a better job at emphasizing the uncertainty while still showing that polls can be pretty reliable.

sixothree 6y ago
It also discounts the penetration of board of elections in some 30+ states.
thinkloop 6y ago
> Giving something like a 95-98% chance to Hillary was arguably a fundamental failure Why was it a fundamental failure? A 5% chance is one in twenty, it happens.
- fabian2k 6y ago
  Im not saying that because of the percentage alone, but because I think the methodology there was suspect. And this was criticized before the election. The lead in the polls was not large, and making it look like a certain thing by treating the states as almost independent results just doesn't make any sense to me.
  - deleted 6y ago
    [deleted]
- richardw 6y ago
  You think if Hillary went up against Trump, America would choose her 19 times out of 20? I’m not sure that makes the mental calculation any better than 95%. 95% is “Obama vs a dog”. Maybe in another country. Every 20 elections, Obama is bitten by the dog and doesn’t make it.
  - thinkloop 6y ago
    > 95% is “Obama vs a dog” That's what it was, don't forget how impossible it seemed at the time, it was a giant upset and shock. > You think if Hillary went up against Trump, America would choose her 19 times out of 20? If you are using "Hillary" and "Trump" figuratively about future elections with similarly matched candidates, then yes, see above.
    - tomp 6y ago
      > That's what it was, don't forget how impossible it seemed at the time, it was a giant upset and shock. Only in your filter bubble.
      - enraged_camel 6y ago
        Nope. Even Trump himself did not expect to win. That's part of why he was utterly unprepared for a transition, as it has been widely documented.
        monocasa 6y ago
        Being extremely charitable in my definitions, Trump seems to heavily apply just in time decision making to nearly all aspects of his life. I wouldn't take his not being prepared as evidence of anything.
    - ghaff 6y ago
      It was an upset/shock. It also wasn't inconceivable or "In an unbelievable upset, the Libertarian Party has won."
    - AnimalMuppet 6y ago
      You're forgetting how bad Hillary was as a candidate. I'm not saying she ran a bad campaign (though she did). I'm talking about her "negatives". Decades of scandals. (Yeah, Trump had them too, but at a minimum it meant that Hillary couldn't use Trump's scandals against him. Also, Hillary's scandals got a lot more national coverage when they happened than Trump's did.) Benghazi. The email server (and with it, the impression that she thought that rules were for other people). The impression that she thought that she was owed the presidency, rather than having to earn it. The way the DNC chose her over Sanders, overruling the will of many of the primary voters. And on and on. It wasn't obvious at the time, because much of the press was pro-Hillary. But she was a terrible candidate. I think if the Democrats had run anyone else, they probably would have won against Trump.
yk 6y ago
In a scientific paper, the author would just write down what they do, and trust the reader to be experienced enough to discount such problems themselves. Now, journalists write for a general audience, that is they write for the lowest common denominator, and accordingly the question what they should write there becomes a quite interesting question. So, 538 did actually change their error estimate during the 2016 campaign to better account for problems of correlation. From a purely mathematical standpoint that is kinda the wrong thing to do, but it is arguably better in line with what the readers expect an error estimate to be.
- sukilot 6y ago
  Why is it wrong?
throwaway2245 6y ago
I'd argue that 538's model likely overestimates uncertainty, because it is a benefit to them both ways. Either they spin it that they were perfectly correct in those 95% of cases where they get it right, or they spin it that they were the least wrong (and therefore the most correct) in the other 5% of cases where everyone gets it wrong.
- frgtpsswrdlame 6y ago
  It's not a benefit to them both ways, it's only useful to them if there's a substantial amount of those 5%s in which case they're probably right that everyone else is underestimating uncertainty.
- ghaff 6y ago
  538 has actually written about calibrating past forecasts. https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/
  - throwaway2245 6y ago
    Their calibration is good in the context of this article, but there is a reason this article doesn't use their Presidential election model - there isn't enough data to do this calibration here.
- chalst 6y ago
  At no point did 538 make such a high forecast: their highest forecast was 88%. There's a joke that you should always express 60% confidence in your predictions, since if the prediction pans out, you can claim to be right, but if it fails, you can bring attention to the "two times out of five wrong" part.
  - throwaway2245 6y ago
    Yes - thanks for explaining this better.
- panarky 6y ago
ghaff 6y ago
In reality, I'm not sure I believe there's such a thing as a 95-98% probability of one major party winning in a country like the US. One can argue whether it's appropriate to fudge in additional uncertainty but there are a lot of things that could happen in the week before the election (including but not limited to candidates dying) that could throw existing poll results up in the air. [ADDED: Or maybe something like that really is a few percent probability and you can end up with a 95% probability anyway. It just feels as if there's some upper limit to what you can measure using polls.]
throwaway082220 6y ago
> Giving something like a 95-98% chance to Hillary was arguably a fundamental failure. Was it? What's the likely hood that someone who was polling as well as Clinton losing? 15% like The Upshot at the NY Times had? ~7% like with Sam Wang's model? Even 7% is around 1 in 14, not something shockingly improbable. People often act like Silver's prediction was good because it gave Trump a higher probability of winning than many of the others, but that's not how probability works. If you say there's a 1/6 chance of rolling a die and getting a 2, and I say there's a 5/6 chance of rolling a 2, and we roll and get a 2, that doesn't mean that I'm correct. I don't think we've had enough rolls of elections resembling 2016 to really have a good grasp of where the percentages should be. In general I question the value of assigning probabilities to election outcomes. This is especially true when you look at the probabilities a few months earlier - for instance, 538 had Clinton going from 49.9% on July 30, 2016 to 88.1% on October 18, 2016. Look at the probabilities they gave during the recent Democratic primaries, and they're also very bouncy. These probabilities lead people to believe that there's a much better understanding of the state of the race than what actually exists, to the point where I'd argue it edges up against pseudoscience.
- caconym_ 6y ago
  Election forecasting is mostly about trying to quantify the current state of play based on imperfect signals. There is theoretically a "right answer" well before the final tally, but without being able to look inside people's heads en masse you can only guess at it. Still, this is conceptually different from forecasting the behavior of a system that's actually subject to randomness or [semi-]chaotic instability, where the given uncertainties will correspond at least partly with actual nondeterminism sitting between the current state of the system and the answer. Therefore I think it's fair to say that the weight an election forecast assigns to the actual winner is a direct indicator of the accuracy of its model. We aren't trying to guess at how a set of dice are weighted, knowing they'll only be thrown once—we're trying to get as close as we can to knowing who is going to vote and who they are going to vote for, and (absent some large disaster or upheaval) a misforecast will be largely attributable to systematic errors in our methodology.
chrisweekly 6y ago
"he" does a better job... who is "he" in your last sentence? ty
seanmcdirmid 6y ago
> Some of the other models for the last election were really bad. Giving something like a 95-98% chance to Hillary was arguably a fundamental failure. The page is still up, you don’t have to pull that number from memory. At the end, it was also nowhere near 95-98%. https://projects.fivethirtyeight.com/2016-election-forecast/ https://projects.fivethirtyeight.com/2016-election-forecast/
- CrazyStat 6y ago
  By "other models" I think he means models other than 538's, several of which did put the probability of Clinton winning at over 95%.
- xapata 6y ago
  Why would estimating a 95% chance for Hillary be a fundamental failure? 5% likelihood things happen often. You've never rolled a 20 on a 20-sided die?
  - patmorgan23 6y ago
    Yep the odds of anyone person getting hit by lightning in a year is 1 in 700,000 yet every year several people are struck.

sixothree 6y ago

It also discounts the penetration of board of elections in some 30+ states.

thinkloop 6y ago

> Giving something like a 95-98% chance to Hillary was arguably a fundamental failure Why was it a fundamental failure? A 5% chance is one in twenty, it happens.

fabian2k 6y ago
Im not saying that because of the percentage alone, but because I think the methodology there was suspect. And this was criticized before the election. The lead in the polls was not large, and making it look like a certain thing by treating the states as almost independent results just doesn't make any sense to me.
- deleted 6y ago
  [deleted]
richardw 6y ago
You think if Hillary went up against Trump, America would choose her 19 times out of 20? I’m not sure that makes the mental calculation any better than 95%. 95% is “Obama vs a dog”. Maybe in another country. Every 20 elections, Obama is bitten by the dog and doesn’t make it.
- thinkloop 6y ago
  > 95% is “Obama vs a dog” That's what it was, don't forget how impossible it seemed at the time, it was a giant upset and shock. > You think if Hillary went up against Trump, America would choose her 19 times out of 20? If you are using "Hillary" and "Trump" figuratively about future elections with similarly matched candidates, then yes, see above.
  - tomp 6y ago
    > That's what it was, don't forget how impossible it seemed at the time, it was a giant upset and shock. Only in your filter bubble.
    - enraged_camel 6y ago
      Nope. Even Trump himself did not expect to win. That's part of why he was utterly unprepared for a transition, as it has been widely documented.
      - monocasa 6y ago
        Being extremely charitable in my definitions, Trump seems to heavily apply just in time decision making to nearly all aspects of his life. I wouldn't take his not being prepared as evidence of anything.
  - ghaff 6y ago
    It was an upset/shock. It also wasn't inconceivable or "In an unbelievable upset, the Libertarian Party has won."
  - AnimalMuppet 6y ago
    You're forgetting how bad Hillary was as a candidate. I'm not saying she ran a bad campaign (though she did). I'm talking about her "negatives". Decades of scandals. (Yeah, Trump had them too, but at a minimum it meant that Hillary couldn't use Trump's scandals against him. Also, Hillary's scandals got a lot more national coverage when they happened than Trump's did.) Benghazi. The email server (and with it, the impression that she thought that rules were for other people). The impression that she thought that she was owed the presidency, rather than having to earn it. The way the DNC chose her over Sanders, overruling the will of many of the primary voters. And on and on. It wasn't obvious at the time, because much of the press was pro-Hillary. But she was a terrible candidate. I think if the Democrats had run anyone else, they probably would have won against Trump.

yk 6y ago

In a scientific paper, the author would just write down what they do, and trust the reader to be experienced enough to discount such problems themselves. Now, journalists write for a general audience, that is they write for the lowest common denominator, and accordingly the question what they should write there becomes a quite interesting question. So, 538 did actually change their error estimate during the 2016 campaign to better account for problems of correlation. From a purely mathematical standpoint that is kinda the wrong thing to do, but it is arguably better in line with what the readers expect an error estimate to be.

sukilot 6y ago
Why is it wrong?

throwaway2245 6y ago

I'd argue that 538's model likely overestimates uncertainty, because it is a benefit to them both ways. Either they spin it that they were perfectly correct in those 95% of cases where they get it right, or they spin it that they were the least wrong (and therefore the most correct) in the other 5% of cases where everyone gets it wrong.

frgtpsswrdlame 6y ago
It's not a benefit to them both ways, it's only useful to them if there's a substantial amount of those 5%s in which case they're probably right that everyone else is underestimating uncertainty.
ghaff 6y ago
538 has actually written about calibrating past forecasts. https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/
- throwaway2245 6y ago
  Their calibration is good in the context of this article, but there is a reason this article doesn't use their Presidential election model - there isn't enough data to do this calibration here.
chalst 6y ago
At no point did 538 make such a high forecast: their highest forecast was 88%. There's a joke that you should always express 60% confidence in your predictions, since if the prediction pans out, you can claim to be right, but if it fails, you can bring attention to the "two times out of five wrong" part.
- throwaway2245 6y ago
  Yes - thanks for explaining this better.
panarky 6y ago

ghaff 6y ago

In reality, I'm not sure I believe there's such a thing as a 95-98% probability of one major party winning in a country like the US. One can argue whether it's appropriate to fudge in additional uncertainty but there are a lot of things that could happen in the week before the election (including but not limited to candidates dying) that could throw existing poll results up in the air. [ADDED: Or maybe something like that really is a few percent probability and you can end up with a 95% probability anyway. It just feels as if there's some upper limit to what you can measure using polls.]

throwaway082220 6y ago

> Giving something like a 95-98% chance to Hillary was arguably a fundamental failure. Was it? What's the likely hood that someone who was polling as well as Clinton losing? 15% like The Upshot at the NY Times had? ~7% like with Sam Wang's model? Even 7% is around 1 in 14, not something shockingly improbable. People often act like Silver's prediction was good because it gave Trump a higher probability of winning than many of the others, but that's not how probability works. If you say there's a 1/6 chance of rolling a die and getting a 2, and I say there's a 5/6 chance of rolling a 2, and we roll and get a 2, that doesn't mean that I'm correct. I don't think we've had enough rolls of elections resembling 2016 to really have a good grasp of where the percentages should be. In general I question the value of assigning probabilities to election outcomes. This is especially true when you look at the probabilities a few months earlier - for instance, 538 had Clinton going from 49.9% on July 30, 2016 to 88.1% on October 18, 2016. Look at the probabilities they gave during the recent Democratic primaries, and they're also very bouncy. These probabilities lead people to believe that there's a much better understanding of the state of the race than what actually exists, to the point where I'd argue it edges up against pseudoscience.

caconym_ 6y ago
Election forecasting is mostly about trying to quantify the current state of play based on imperfect signals. There is theoretically a "right answer" well before the final tally, but without being able to look inside people's heads en masse you can only guess at it. Still, this is conceptually different from forecasting the behavior of a system that's actually subject to randomness or [semi-]chaotic instability, where the given uncertainties will correspond at least partly with actual nondeterminism sitting between the current state of the system and the answer. Therefore I think it's fair to say that the weight an election forecast assigns to the actual winner is a direct indicator of the accuracy of its model. We aren't trying to guess at how a set of dice are weighted, knowing they'll only be thrown once—we're trying to get as close as we can to knowing who is going to vote and who they are going to vote for, and (absent some large disaster or upheaval) a misforecast will be largely attributable to systematic errors in our methodology.

chrisweekly 6y ago

"he" does a better job... who is "he" in your last sentence? ty

seanmcdirmid 6y ago

> Some of the other models for the last election were really bad. Giving something like a 95-98% chance to Hillary was arguably a fundamental failure. The page is still up, you don’t have to pull that number from memory. At the end, it was also nowhere near 95-98%. https://projects.fivethirtyeight.com/2016-election-forecast/ https://projects.fivethirtyeight.com/2016-election-forecast/

CrazyStat 6y ago
By "other models" I think he means models other than 538's, several of which did put the probability of Clinton winning at over 95%.
xapata 6y ago
Why would estimating a 95% chance for Hillary be a fundamental failure? 5% likelihood things happen often. You've never rolled a 20 on a 20-sided die?
- patmorgan23 6y ago
  Yep the odds of anyone person getting hit by lightning in a year is 1 in 700,000 yet every year several people are struck.