4 ms·

The article seems like an oversimplification of the underlying dispute. - It’s not necessary or even useful to define an arbitrary “decision boundary” unless y

by comex 8y ago

The article seems like an oversimplification of the underlying dispute. - It’s not necessary or even useful to define an arbitrary “decision boundary” unless you actually have to make a decision. From a Bayesian perspective, a probability stands for itself: 100% means the event is certain to occur, 50% means you have no idea, and numbers in between convey varying levels of certainty. In reality, 538’s predictions are not true Bayesian probabilities because they don’t take epistemic uncertainty into account, but that’s a totally different issue. - “Wild fluctuations in a prediction” from new information are absolutely a “normal part of forecasting” - sometimes. If I’m planning to flip two coins, the probability of getting two heads is 25%; but once I flip the first coin, the probability changes to either 50% (if I get heads) or 0% (if I get tails). In the case of Comey reopening the investigation, even if the model had included a probability of that happening, it would have been low and thus wouldn’t affect the overall forecast much. But once that low probability became a certainty, you would expect a sudden swing. The real question is whether 538’s predictions are more swingy than they logically should be (particularly earlier on), but again, that’s a different issue.

kingkawn 8y ago
Seems like in addition to whether they're more swingy than they should be, are they being presented in a way that is interpreted as more certain or authoritative than they can possibly claim?
- c3534l 8y ago
  538 provides pretty in-depth discussions of it forecasts and changes in it. Far more than you would expect - and I think because 538 is ultimately a statistics blog at heart, not a political blog.
- azernik 8y ago
  That is actually an issue they've acknowledged and tried to address. People also tended to see percentages and think they reflected vote percents. To address this, for the 2018 election cycle 538 started displaying their odds as numerical ratios (5 in 9, for example) to try to reduce the aura of certainty.
  - dagw 8y ago
    538 started displaying their odds as numerical ratios Did that actually make things clearer to people? Speaking only for myself, I find ratios quite hard to works with and reason about. I certainly couldn't tell you instantly for example if 5/9 is more or less than 4/7 or how big the difference between the two are without first doing some mental arithmetic. Perhaps people in the US are more used to working with ratios since their measurement systems tend to be ratio based.
    - nscalf 8y ago
      That's the point though. It's meant to require a bit of thinking to digest the information. They didn't set out to make the information clearer and easier to understand, they set out to make the false positive of reading a percentage and considering it final results less frequent.
- VintageCool 8y ago
  The 2016 election was swingy and it was uncertain. In 2008 and 2012, the political media was myopically focused on the twists and turns of the horse race and called it 50-50 dead even. Nate Silver's 538 modeled the election as basically stable with only small polling shifts, and the only change in the last month was a steady decline in the remaining time that McCain (or Romney) had to significantly shift the polls. In 2016, the political media treated the election as if Hillary had a lock on it, and entreated us not to get swept up in the daily swings in the polls. Meanwhile, Nate Silver's model swung wildly with the polls, and Nate went on the media to emphasize the uncertainty of the election. Both candidates were disliked, and a large portion of the electorate was undecided right up until the last week of the election. I remember the following exchange in one interview: Interviewer: If you say that Donald Trump has a 33% chance to win, what's the chance that he actually wins? Nate Silver: One in three. I'm predicting a one in three chance that Donald Trump wins in November. As for the authoritativeness of their presentation, FiveThirtyEight has a really beautiful forecast that presents their 2018 prediction as a probability distribution: https://projects.fivethirtyeight.com/2018-midterm-election-forecast/house/ https://projects.fivethirtyeight.com/2018-midterm-election-f...
  - paganel 8y ago
    >The 2016 election was swingy and it was uncertain. I don't remember it that way, in fact I remember seeing a lot of very shocked faces in the Democratic camp when the results were beginning to take shape. I don't remember having seen similar confused reactions after any previous US presidential elections, not even after the Bush vs Gore one which was a lot closer in terms of electoral votes. Back to the article, I think Nate Silver's failure only shows to the general public that electoral predictions are rubbish. Maybe "failure" is a strong word because he genuinely seems to be the best at what he's doing, it's just that the domain in which he's involved is turning out to be bogus. I'm sure that there was a crystal-ball viewer that was the best at what he/she was doing, it's just that crystal-ball viewing turned out to be bogus.
    - FactolSarin 8y ago
      However, FiveThirtyEight gave Trump a 30% chance of winning, and in the days before the election was stressing the fact that a Trump win was very possible. They didn't predict his win, but they did better than anyone else in the mainstream.
chimeracoder 8y ago
> 100% means the event is certain to occur Actually, 100% means the event is "almost certain". That's not an accidental choice of words; it's a technical term which conveys important meaning about how we reason about probability.
- comex 8y ago
  I had to look this up, so for the benefit of anyone else curious: https://en.wikipedia.org/wiki/Almost_surely https://en.wikipedia.org/wiki/Almost_surely TL;DR: Infinities being weird as usual.
  - tempestn 8y ago
    Good read. From that page, here's an example that nicely illustrates the concept: Imagine throwing a dart at a unit square (i.e. a square with area 1) so that the dart always hits exactly one point of the square, and so that each point in the square is equally likely to be hit. Now, notice that since the square has area 1, the probability that the dart will hit any particular subregion of the square equals the area of that subregion. For example, the probability that the dart will hit the right half of the square is 0.5, since the right half has area 0.5. Next, consider the event that "the dart hits a diagonal of the unit square exactly". Since the areas of the diagonals of the square are zero, the probability that the dart lands exactly on a diagonal is zero. So, the dart will almost never land on a diagonal (i.e. it will almost surely not land on a diagonal). Nonetheless the set of points on the diagonals is not empty and a point on a diagonal is no less possible than any other point: the diagonal does contain valid outcomes of the experiment.
    - simonh 8y ago
      Interesting. Of course to do the experiment as described you would have to determine the point the dart hit with infinite precision. If there is any limit to the precision with which you can determine the point of impact, then the probability that the dart hits close enough to the diagonal that you can't tell if it was hit or not becomes finite.
- 8y ago

by comex 8y ago

kingkawn 8y ago

Seems like in addition to whether they're more swingy than they should be, are they being presented in a way that is interpreted as more certain or authoritative than they can possibly claim?

c3534l 8y ago
538 provides pretty in-depth discussions of it forecasts and changes in it. Far more than you would expect - and I think because 538 is ultimately a statistics blog at heart, not a political blog.
azernik 8y ago
That is actually an issue they've acknowledged and tried to address. People also tended to see percentages and think they reflected vote percents. To address this, for the 2018 election cycle 538 started displaying their odds as numerical ratios (5 in 9, for example) to try to reduce the aura of certainty.
- dagw 8y ago
  538 started displaying their odds as numerical ratios Did that actually make things clearer to people? Speaking only for myself, I find ratios quite hard to works with and reason about. I certainly couldn't tell you instantly for example if 5/9 is more or less than 4/7 or how big the difference between the two are without first doing some mental arithmetic. Perhaps people in the US are more used to working with ratios since their measurement systems tend to be ratio based.
  - nscalf 8y ago
    That's the point though. It's meant to require a bit of thinking to digest the information. They didn't set out to make the information clearer and easier to understand, they set out to make the false positive of reading a percentage and considering it final results less frequent.
VintageCool 8y ago
The 2016 election was swingy and it was uncertain. In 2008 and 2012, the political media was myopically focused on the twists and turns of the horse race and called it 50-50 dead even. Nate Silver's 538 modeled the election as basically stable with only small polling shifts, and the only change in the last month was a steady decline in the remaining time that McCain (or Romney) had to significantly shift the polls. In 2016, the political media treated the election as if Hillary had a lock on it, and entreated us not to get swept up in the daily swings in the polls. Meanwhile, Nate Silver's model swung wildly with the polls, and Nate went on the media to emphasize the uncertainty of the election. Both candidates were disliked, and a large portion of the electorate was undecided right up until the last week of the election. I remember the following exchange in one interview: Interviewer: If you say that Donald Trump has a 33% chance to win, what's the chance that he actually wins? Nate Silver: One in three. I'm predicting a one in three chance that Donald Trump wins in November. As for the authoritativeness of their presentation, FiveThirtyEight has a really beautiful forecast that presents their 2018 prediction as a probability distribution: https://projects.fivethirtyeight.com/2018-midterm-election-forecast/house/ https://projects.fivethirtyeight.com/2018-midterm-election-f...
- paganel 8y ago
  >The 2016 election was swingy and it was uncertain. I don't remember it that way, in fact I remember seeing a lot of very shocked faces in the Democratic camp when the results were beginning to take shape. I don't remember having seen similar confused reactions after any previous US presidential elections, not even after the Bush vs Gore one which was a lot closer in terms of electoral votes. Back to the article, I think Nate Silver's failure only shows to the general public that electoral predictions are rubbish. Maybe "failure" is a strong word because he genuinely seems to be the best at what he's doing, it's just that the domain in which he's involved is turning out to be bogus. I'm sure that there was a crystal-ball viewer that was the best at what he/she was doing, it's just that crystal-ball viewing turned out to be bogus.
  - FactolSarin 8y ago
    However, FiveThirtyEight gave Trump a 30% chance of winning, and in the days before the election was stressing the fact that a Trump win was very possible. They didn't predict his win, but they did better than anyone else in the mainstream.

chimeracoder 8y ago

> 100% means the event is certain to occur Actually, 100% means the event is "almost certain". That's not an accidental choice of words; it's a technical term which conveys important meaning about how we reason about probability.

comex 8y ago
I had to look this up, so for the benefit of anyone else curious: https://en.wikipedia.org/wiki/Almost_surely https://en.wikipedia.org/wiki/Almost_surely TL;DR: Infinities being weird as usual.
- tempestn 8y ago
  Good read. From that page, here's an example that nicely illustrates the concept: Imagine throwing a dart at a unit square (i.e. a square with area 1) so that the dart always hits exactly one point of the square, and so that each point in the square is equally likely to be hit. Now, notice that since the square has area 1, the probability that the dart will hit any particular subregion of the square equals the area of that subregion. For example, the probability that the dart will hit the right half of the square is 0.5, since the right half has area 0.5. Next, consider the event that "the dart hits a diagonal of the unit square exactly". Since the areas of the diagonals of the square are zero, the probability that the dart lands exactly on a diagonal is zero. So, the dart will almost never land on a diagonal (i.e. it will almost surely not land on a diagonal). Nonetheless the set of points on the diagonals is not empty and a point on a diagonal is no less possible than any other point: the diagonal does contain valid outcomes of the experiment.
  - simonh 8y ago
    Interesting. Of course to do the experiment as described you would have to determine the point the dart hit with infinite precision. If there is any limit to the precision with which you can determine the point of impact, then the probability that the dart hits close enough to the diagonal that you can't tell if it was hit or not becomes finite.
8y ago