4 ms·

I would ask instead: what does it even mean to be "right" about a statistical probability of a future event? What does 54% chance mean vs an 84% chance when you

I would ask instead: what does it even mean to be "right" about a statistical probability of a future event? What does 54% chance mean vs an 84% chance when you're talking about something that's six weeks away? If the event happens or doesn't, that fact in itself does not tell you whether 54 or 84 was the right number. Each group creating predictions builds a mathematical model that simplifies reality into a set of numbers that necessarily have large uncertainty and error built in. The models are, by design, not intended to replicate reality. They are intended to be a smaller, simplified version, retaining as many of the relevant properties as possible, but with no illusions (one hopes) that they actually represent reality. (If they did, you'd expect them to make exact predictions, not percentage chances.) Presumably these teams do their own math "right", so the only useful question is "which model is _closer_ to reality?" But there's no way to know that, except to test them an infinite number of times, and ooops can't do that either...

grandalf 10y ago
I'm very glad someone made this point.
platz 10y ago
It is very simple. All one has to do is imagine the election being repeated an infinity number of times under a stationary-non-changing distribution.
- chimeracoder 10y ago
  > It is very simple. It's actually not simple at all! This has been one of the most contentious questions of statistics for over a century. > All one has to do is imagine the election being repeated an infinity number of times under a stationary-non-changing distribution. That's using a frequentist interpretation of probability, which is not the only interpretation. (Nor is it, in fact, the interpretation used by e.g. Nate Silver, who is one of the statisticians mentioned in the article). (The frequentist interpretation, incidentally, has been shown to be self-contradictory, for reasons that involve continuous mathematics that I won't get into. In other words, you can construct situations that have nonsensical interpretations under the frequentist approach. That doesn't mean frequentist statistics aren't useful, but - like any model - the approach has limitations.)
  - platz 10y ago
    Yes, my hope was to point out the absurdity of this situation for an event that only happens once. It would be nice if all polls calculated a CI or probability distribution of their results (i know 538 does). But even 538 doesn't display their priors. But this would confuse the public even more.. e... asking them to interpret a 95% chance to win with low confidence
- deleted 10y ago
  [deleted]
- defen 10y ago
  I realize you're joking, but if the election were repeated an infinity number of times, wouldn't the same candidate win each time?
  - platz 10y ago
    The distribution for the roll of a die doesn't change, but the individual outcomes do. It all boils down to how this random variable is defined.
    - defen 10y ago
      Right, but what mechanism would change the outcome of the election if it were re-run? Are you suggesting that some percentage of people randomly choose their candidate while they are in the voting booth?
      - platz 10y ago
        I know I do!
  - chimeracoder 10y ago
    > if the election were repeated an infinity number of times, wouldn't the same candidate win each time? It depends on what you mean by "repeated an infinite number of times". First, there's a problem with the framing of that statement, because you can't literally repeat something an infinite number of times. You can repeat something indefinitely, and observe the behavior as the number of repetitions increases without bound, but that's not quite the same thing. (I realize this sounds pedantic, but if you want to understand the underlying interpretation of probability, it's an important distinction to be aware of). So, let's say you asked, "if the election were repeated over and over again, without bound, what would the outcomes be?" And the answer to that is... well, it depends on who you ask. Setting aside the "fate vs. free will" philosophical question, different statisticians would answer that question differently, depending on which interpretation of probability they use. There are three main interpretations of probability, though these days, people pretty much only ever use two (frequentist and Bayesian). And you could construct either a frequentist or a Bayesian argument that the same candidate would always win, or that a different candidate could win each time. Think of the election as a function - some unknown inputs go in, and an outcome goes out. What are the inputs to that function? (That's a rhetorical question - we're not really sure what makes every individual voter vote, and what disruptions could happen to nudge each person to vote a different way, or to not make it to the polls that day, or for their ballot to get lost in the mail, etc.). Congratulations, you have a random variable[0]! Now all you have to do is decompose it to determine the underlying components and the true distribution they draw from[1]. I realize that's not a very satisfying answer, but the underlying issue ("what does probability mean") is not a simple question, and statisticians have been debating it for over a century. [0] which is, confusingly, neither random nor a variable. [1] Which is itself controversial way of putting it, because some statisticians would argue that there is no "true" distribution, or that, if there is, we can never actually know it.
smallnamespace 10y ago
People haven't actually agreed on a single definition for what 'probability' means, although we have various definitions that often lead to the same numbers in practical cases [1]. [1] https://en.wikipedia.org/wiki/Probability_interpretations#Philosophy https://en.wikipedia.org/wiki/Probability_interpretations#Ph...
elicash 10y ago
I'm no expert here. But my understanding is that while we're using them to predict behavior, what's ACTUALLY happening is that numbers are describing what's happened in previous elections when there were certain economic or polling numbers. So maybe the issue is just in the language we use to talk about it, that we shouldn't even refer to it as forecasting at all.
- inimino 10y ago
  It's because certain things stay the same between elections that the forecast, while based on the past, has some validity. And it is because every historical situation is different that we can have lively speculation about whether the election is historically unprecedented in some way that invalidates the forecast.
mcshicks 10y ago
There is a website (good judgement project) where you can compete with other players making probability predictions for actual events. They use something called a "brier score" to measure your predicative ability against others. You are right that you need an infinite set to know, but with a large set of predictions you can compare predicted frequencies with actual. https://en.wikipedia.org/wiki/Brier_score https://en.wikipedia.org/wiki/Brier_score
pwinnski 10y ago
It makes a nice headline to reduce the prediction to "a 54% chance vs an 84% chance," but most of these sites are predicting not just the statistical probability of a future event in "happens or doesn't" form, they're predicting the relative vote percentage for each candidate in each state. So if Trump beats Clinton, that really embarrassing for 538, but if she wins the popular vote by less than 6.3%, that's also slightly off. If he wins Florida, that's slightly less off, but still a miss for 538, and so on. IMO, the only way to really know which model is closer to reality is to wait until November 9 and find out. That's not helpful for deciding who to pay attention to in 2010, though.

grandalf 10y ago

I'm very glad someone made this point.

platz 10y ago

It is very simple. All one has to do is imagine the election being repeated an infinity number of times under a stationary-non-changing distribution.

chimeracoder 10y ago
> It is very simple. It's actually not simple at all! This has been one of the most contentious questions of statistics for over a century. > All one has to do is imagine the election being repeated an infinity number of times under a stationary-non-changing distribution. That's using a frequentist interpretation of probability, which is not the only interpretation. (Nor is it, in fact, the interpretation used by e.g. Nate Silver, who is one of the statisticians mentioned in the article). (The frequentist interpretation, incidentally, has been shown to be self-contradictory, for reasons that involve continuous mathematics that I won't get into. In other words, you can construct situations that have nonsensical interpretations under the frequentist approach. That doesn't mean frequentist statistics aren't useful, but - like any model - the approach has limitations.)
- platz 10y ago
  Yes, my hope was to point out the absurdity of this situation for an event that only happens once. It would be nice if all polls calculated a CI or probability distribution of their results (i know 538 does). But even 538 doesn't display their priors. But this would confuse the public even more.. e... asking them to interpret a 95% chance to win with low confidence
deleted 10y ago
[deleted]
defen 10y ago
I realize you're joking, but if the election were repeated an infinity number of times, wouldn't the same candidate win each time?
- platz 10y ago
  The distribution for the roll of a die doesn't change, but the individual outcomes do. It all boils down to how this random variable is defined.
  - defen 10y ago
    Right, but what mechanism would change the outcome of the election if it were re-run? Are you suggesting that some percentage of people randomly choose their candidate while they are in the voting booth?
    - platz 10y ago
      I know I do!
- chimeracoder 10y ago
  > if the election were repeated an infinity number of times, wouldn't the same candidate win each time? It depends on what you mean by "repeated an infinite number of times". First, there's a problem with the framing of that statement, because you can't literally repeat something an infinite number of times. You can repeat something indefinitely, and observe the behavior as the number of repetitions increases without bound, but that's not quite the same thing. (I realize this sounds pedantic, but if you want to understand the underlying interpretation of probability, it's an important distinction to be aware of). So, let's say you asked, "if the election were repeated over and over again, without bound, what would the outcomes be?" And the answer to that is... well, it depends on who you ask. Setting aside the "fate vs. free will" philosophical question, different statisticians would answer that question differently, depending on which interpretation of probability they use. There are three main interpretations of probability, though these days, people pretty much only ever use two (frequentist and Bayesian). And you could construct either a frequentist or a Bayesian argument that the same candidate would always win, or that a different candidate could win each time. Think of the election as a function - some unknown inputs go in, and an outcome goes out. What are the inputs to that function? (That's a rhetorical question - we're not really sure what makes every individual voter vote, and what disruptions could happen to nudge each person to vote a different way, or to not make it to the polls that day, or for their ballot to get lost in the mail, etc.). Congratulations, you have a random variable[0]! Now all you have to do is decompose it to determine the underlying components and the true distribution they draw from[1]. I realize that's not a very satisfying answer, but the underlying issue ("what does probability mean") is not a simple question, and statisticians have been debating it for over a century. [0] which is, confusingly, neither random nor a variable. [1] Which is itself controversial way of putting it, because some statisticians would argue that there is no "true" distribution, or that, if there is, we can never actually know it.

smallnamespace 10y ago

People haven't actually agreed on a single definition for what 'probability' means, although we have various definitions that often lead to the same numbers in practical cases [1]. [1] https://en.wikipedia.org/wiki/Probability_interpretations#Philosophy https://en.wikipedia.org/wiki/Probability_interpretations#Ph...

elicash 10y ago

I'm no expert here. But my understanding is that while we're using them to predict behavior, what's ACTUALLY happening is that numbers are describing what's happened in previous elections when there were certain economic or polling numbers. So maybe the issue is just in the language we use to talk about it, that we shouldn't even refer to it as forecasting at all.

inimino 10y ago
It's because certain things stay the same between elections that the forecast, while based on the past, has some validity. And it is because every historical situation is different that we can have lively speculation about whether the election is historically unprecedented in some way that invalidates the forecast.

mcshicks 10y ago

There is a website (good judgement project) where you can compete with other players making probability predictions for actual events. They use something called a "brier score" to measure your predicative ability against others. You are right that you need an infinite set to know, but with a large set of predictions you can compare predicted frequencies with actual. https://en.wikipedia.org/wiki/Brier_score https://en.wikipedia.org/wiki/Brier_score

pwinnski 10y ago

It makes a nice headline to reduce the prediction to "a 54% chance vs an 84% chance," but most of these sites are predicting not just the statistical probability of a future event in "happens or doesn't" form, they're predicting the relative vote percentage for each candidate in each state. So if Trump beats Clinton, that really embarrassing for 538, but if she wins the popular vote by less than 6.3%, that's also slightly off. If he wins Florida, that's slightly less off, but still a miss for 538, and so on. IMO, the only way to really know which model is closer to reality is to wait until November 9 and find out. That's not helpful for deciding who to pay attention to in 2010, though.