5 ms·

>Hillary Clinton had a >70% of winning the US presidential election Probabilities without confidence intervals[1] are by-and-large meaningless (She has a 90% c

>Hillary Clinton had a >70% of winning the US presidential election Probabilities without confidence intervals[1] are by-and-large meaningless (She has a 90% chance of of winning with a confidence interval of +11% -100%). No amount of d3.js on 538's blog will change this. https://en.wikipedia.org/wiki/Confidence_interval https://en.wikipedia.org/wiki/Confidence_interval

OscarCunningham 8y ago
From a Bayesian perspective, or from a betting one, it doesn't make sense to put probabilities in a confidence interval. You might be uncertain about the world, but you can be certain about how much uncertainty you have, since it's a property of your own mind.
- fragsworth 8y ago
  How do we quantify the difference between coin tosses, which we are very certain is 50% likely to end up heads, from political elections, where we only have a few examples to go off of?
  - edbaskerville 8y ago
    In terms of predicting what's going to happen next, it really isn't that different if your model's any good. Looking backwards, if the 2016 election had been determined by two coin tosses both coming up heads, would we be surprised? And yet 538's polling model indicated that the probability was even better than two coins coming up heads. They probably should have just written that right there: "71%, better odds than two coins coming up heads." But: in order to understand if the model is any good, you need to try to build a good model, describe your model, and describe what you know about the underlying processes. With words, visuals, and math. Which is pretty much what 538 does better than most.
  - OscarCunningham 8y ago
    Good question. The difference is how the probability changes in response to new information. Learning more about the coin won't change the probability of heads from 50%, but doing more exhaustive polls would have likely improved our prediction about the election. However I don't think there's as much of a difference as you think. If we learnt more about how the coin was going to be flipped then that would certainly improve our estimate of the outcome. If we found out the exact way it would be flipped we could calculate the outcome exactly. So we can't really compare the two situations quantitatively, since there's no way to match up like-for-like the information we could receive. But we can say, for each possible piece of information we could receive, how much we expect it to change our probability.
- edbaskerville 8y ago
  This is worth emphasizing. If you have a model with a parameter that represents the probability of something happening, and then you have uncertainty in that parameter, your real probability needs to integrate over that uncertainty. But the final probability is still just a single number. But I suspect the GP's reasoning was getting interference from the valid point that poll numbers, which are not probabilities, are meaningless without confidence intervals or some other measure of uncertainty. That uncertainty will then feed, via some model, into a probability estimate of a candidate winning. And that's just a single number.
  - 3pt14159 8y ago
    Hmm. Doesn't it depend on whether re-rolls are allowed as to how confident we can be? For example, I can imagine a well engineered coin that has a 0.5 probability of heads with a 0.005 confidence (ie, we suspect that the true weighting between heads and tails is likely between 0.495 and 0.505 19 times out of 20) and I can contrast it with a hastily made coin that can still have a 0.5 expected probability with a 0.1 confidence (ie, we expect the true probability to be between 0.4 to 0.6, 19 times out of 20). Coin A and Coin B have dramatically different impacts on our decision making. For example, selling insurance against 5 identical flips in a row is a much more expensive proposition for Coin B than Coin A. I feel like I really should know this given my data science background, but sometimes the basics slip away.
    - OscarCunningham 8y ago
      Right. Both coins have a probability of 0.5 on the next flip, but when you do multiple flips the hastily made coin has a higher chance of a result with lots of heads or lots of tails. We could do the same thing for the 2016 election, but we would have to specify exactly what we meant by a "repeat". Do we just let Hillary run for the 2020 election and see what happens? Or do we put back every atom to the exact position it had in 2014, so that the only divergences between the two elections are caused by quantum randomness? Or something in between, like looking at all elections where a demagogue outsider runs against an established insider? Really it doesn't matter what definition of "repeat" we choose. Since the repeat won't actually happen, we can't be called to bet on it, so knowing the probability in that case isn't too useful. Whereas a coin actually can be flipped multiple times.
- TTPrograms 8y ago
  It might be of some general interest to note that there exist such objects - distributions over distributions, or metaprobabilities. They have practical use - for example, if you are playing a game with uncertainty, you might observe something in the next step that changes your belief, or distribution over world state, b = P(s). If you have some expected distribution of next observations then you can talk about the probability of having some belief in the next state. Roughly P(b'|b) = P(P(s')|P(s)). You can collapse this into simple probabilities if you just care about your new observations, P(obs'|b) = \sum_s P(obs'|s)P(s|b). I could imagine some frameworks where confidence intervals in this way would be useful - ex. I have 3 fairly different world models that I think are equally likely, each has a P(election), what's the P(election|world) and get confidence intervals across world models rather than just summing over them to P(election). But I agree that for most common approaches that simple probabilities are most useful and clear.
  - OscarCunningham 8y ago
    Something similar certainly is useful in the case of flipping a coin. Suppose we have two coins, one of which we know is fair and one of which we know is biased but we don't know which way. Then our knowledge about the bias of the first coin would be described by a distribution with a sharp peak around 0.5, but the distribution for the second coin would have two lumps either side of 0.5. Then the probability of heads on the next flip would be 0.5 for either coin. But if someone asked us to bet on the probability of heads on the next two flips, then the second coin would have a higher probability than the first. The different distributions for the two coins don't make a difference for one flip, but do for multiple flips. But there's no meaningful way to repeat an election, so I don't think similar distributions or confidence intervals are useful in that case. EDIT: We can imagine rerolling an election, but since we can't actually do it we don't have to bet on it so the information wouldn't be very useful to us.
- AlexCoventry 8y ago
  > From a Bayesian perspective, or from a betting one, it doesn't make sense to put probabilities in a confidence interval "Bayesian perspective" covers a lot of territory, and your assertion depends on the situation and the modeling objective. If a probability is a model parameter (for instance, frequency of heads for a particular coin), then summarizing the posterior distribution on that parameter with a confidence interval can be a sensible thing to do. Or do you mean as opposed to a credibility interval?
computerphage 8y ago
> Probabilities without confidence intervals are by-and-large meaningless. That's just not true. If I believe that my team has a 20% chance to win and you offer me a bet with anything better than 5-to-1 odds I should take the bet. If you offer me anything worse than 5-to-1, then I should not take the bet. There's no fuzz factor necessary; no confidence interval that I need to use to make the decision. Perhaps you're getting at the idea of calibration? That it's difficult for a person to know what a 20% chance feels like? But there are still a lot of situations where it's not up to human judgement.
- williamscales 8y ago
  I think this is getting to the root of the problem — you're taking a perfectly valid frequentist view of probability, that is, viewing it as a series of discrete experiments. But the probabilities that were assigned to Clinton's victory were derived from Bayesian probability theory that estimates the likelihood that event occurs based on empirically determined prior probabilities that contribute to that event. Those prior distributions have uncertainty which leads to uncertainty in the resulting prediction. I apologize if I've explained this poorly, this is just my layman's understanding of the matter.
  - marcosdumay 8y ago
    It's a property of the thing being measured, not of the measuring system. A Bayesian calculating that probability must get a single number too, without error margins. The only difference is that the Bayesian will have to weight his probabilities by how much of the interval is at the "win" and the "no win" scenarios. It is different if you are measuring how many votes each candidate will have. For that both methods must get intervals and a confidence level.
- IanCal 8y ago
  Perhaps a good way of putting it would be that there's probably not an extremely solid line like that 20%. You would leap at a bet with 10000-1 odds, and never go for 2-1. Let's say you can make these bets repeatedly. Would you always bet on 5.01-1 odds and never on 4.99-1 odds? Is it that odd to think that your team has about a 20% chance to win?
airza 8y ago
What? Confidence intervals are used to assign a confidence to a measure of a larger population.
asfasgasg 8y ago
I've been confused by this in the past too, so let me clarify for you. Confidence intervals are for predictions of a given value. They are not for probabilities. It doesn't make sense to say, "I think she has a 70% chance of winning, plus or minus 5%." On the other hand, it does make sense to say, I think she will get 48% of the vote, with an interval of +-1%.
scythe 8y ago
"Speaking of unknown probabilities or of probability of a probability must be forbidden as meaningless. " - Bruno de Finetti, 1977

OscarCunningham 8y ago

From a Bayesian perspective, or from a betting one, it doesn't make sense to put probabilities in a confidence interval. You might be uncertain about the world, but you can be certain about how much uncertainty you have, since it's a property of your own mind.

fragsworth 8y ago
How do we quantify the difference between coin tosses, which we are very certain is 50% likely to end up heads, from political elections, where we only have a few examples to go off of?
- edbaskerville 8y ago
  In terms of predicting what's going to happen next, it really isn't that different if your model's any good. Looking backwards, if the 2016 election had been determined by two coin tosses both coming up heads, would we be surprised? And yet 538's polling model indicated that the probability was even better than two coins coming up heads. They probably should have just written that right there: "71%, better odds than two coins coming up heads." But: in order to understand if the model is any good, you need to try to build a good model, describe your model, and describe what you know about the underlying processes. With words, visuals, and math. Which is pretty much what 538 does better than most.
- OscarCunningham 8y ago
  Good question. The difference is how the probability changes in response to new information. Learning more about the coin won't change the probability of heads from 50%, but doing more exhaustive polls would have likely improved our prediction about the election. However I don't think there's as much of a difference as you think. If we learnt more about how the coin was going to be flipped then that would certainly improve our estimate of the outcome. If we found out the exact way it would be flipped we could calculate the outcome exactly. So we can't really compare the two situations quantitatively, since there's no way to match up like-for-like the information we could receive. But we can say, for each possible piece of information we could receive, how much we expect it to change our probability.
edbaskerville 8y ago
This is worth emphasizing. If you have a model with a parameter that represents the probability of something happening, and then you have uncertainty in that parameter, your real probability needs to integrate over that uncertainty. But the final probability is still just a single number. But I suspect the GP's reasoning was getting interference from the valid point that poll numbers, which are not probabilities, are meaningless without confidence intervals or some other measure of uncertainty. That uncertainty will then feed, via some model, into a probability estimate of a candidate winning. And that's just a single number.
- 3pt14159 8y ago
  Hmm. Doesn't it depend on whether re-rolls are allowed as to how confident we can be? For example, I can imagine a well engineered coin that has a 0.5 probability of heads with a 0.005 confidence (ie, we suspect that the true weighting between heads and tails is likely between 0.495 and 0.505 19 times out of 20) and I can contrast it with a hastily made coin that can still have a 0.5 expected probability with a 0.1 confidence (ie, we expect the true probability to be between 0.4 to 0.6, 19 times out of 20). Coin A and Coin B have dramatically different impacts on our decision making. For example, selling insurance against 5 identical flips in a row is a much more expensive proposition for Coin B than Coin A. I feel like I really should know this given my data science background, but sometimes the basics slip away.
  - OscarCunningham 8y ago
    Right. Both coins have a probability of 0.5 on the next flip, but when you do multiple flips the hastily made coin has a higher chance of a result with lots of heads or lots of tails. We could do the same thing for the 2016 election, but we would have to specify exactly what we meant by a "repeat". Do we just let Hillary run for the 2020 election and see what happens? Or do we put back every atom to the exact position it had in 2014, so that the only divergences between the two elections are caused by quantum randomness? Or something in between, like looking at all elections where a demagogue outsider runs against an established insider? Really it doesn't matter what definition of "repeat" we choose. Since the repeat won't actually happen, we can't be called to bet on it, so knowing the probability in that case isn't too useful. Whereas a coin actually can be flipped multiple times.
TTPrograms 8y ago
It might be of some general interest to note that there exist such objects - distributions over distributions, or metaprobabilities. They have practical use - for example, if you are playing a game with uncertainty, you might observe something in the next step that changes your belief, or distribution over world state, b = P(s). If you have some expected distribution of next observations then you can talk about the probability of having some belief in the next state. Roughly P(b'|b) = P(P(s')|P(s)). You can collapse this into simple probabilities if you just care about your new observations, P(obs'|b) = \sum_s P(obs'|s)P(s|b). I could imagine some frameworks where confidence intervals in this way would be useful - ex. I have 3 fairly different world models that I think are equally likely, each has a P(election), what's the P(election|world) and get confidence intervals across world models rather than just summing over them to P(election). But I agree that for most common approaches that simple probabilities are most useful and clear.
- OscarCunningham 8y ago
  Something similar certainly is useful in the case of flipping a coin. Suppose we have two coins, one of which we know is fair and one of which we know is biased but we don't know which way. Then our knowledge about the bias of the first coin would be described by a distribution with a sharp peak around 0.5, but the distribution for the second coin would have two lumps either side of 0.5. Then the probability of heads on the next flip would be 0.5 for either coin. But if someone asked us to bet on the probability of heads on the next two flips, then the second coin would have a higher probability than the first. The different distributions for the two coins don't make a difference for one flip, but do for multiple flips. But there's no meaningful way to repeat an election, so I don't think similar distributions or confidence intervals are useful in that case. EDIT: We can imagine rerolling an election, but since we can't actually do it we don't have to bet on it so the information wouldn't be very useful to us.
AlexCoventry 8y ago
> From a Bayesian perspective, or from a betting one, it doesn't make sense to put probabilities in a confidence interval "Bayesian perspective" covers a lot of territory, and your assertion depends on the situation and the modeling objective. If a probability is a model parameter (for instance, frequency of heads for a particular coin), then summarizing the posterior distribution on that parameter with a confidence interval can be a sensible thing to do. Or do you mean as opposed to a credibility interval?

computerphage 8y ago

> Probabilities without confidence intervals are by-and-large meaningless. That's just not true. If I believe that my team has a 20% chance to win and you offer me a bet with anything better than 5-to-1 odds I should take the bet. If you offer me anything worse than 5-to-1, then I should not take the bet. There's no fuzz factor necessary; no confidence interval that I need to use to make the decision. Perhaps you're getting at the idea of calibration? That it's difficult for a person to know what a 20% chance feels like? But there are still a lot of situations where it's not up to human judgement.

williamscales 8y ago
I think this is getting to the root of the problem — you're taking a perfectly valid frequentist view of probability, that is, viewing it as a series of discrete experiments. But the probabilities that were assigned to Clinton's victory were derived from Bayesian probability theory that estimates the likelihood that event occurs based on empirically determined prior probabilities that contribute to that event. Those prior distributions have uncertainty which leads to uncertainty in the resulting prediction. I apologize if I've explained this poorly, this is just my layman's understanding of the matter.
- marcosdumay 8y ago
  It's a property of the thing being measured, not of the measuring system. A Bayesian calculating that probability must get a single number too, without error margins. The only difference is that the Bayesian will have to weight his probabilities by how much of the interval is at the "win" and the "no win" scenarios. It is different if you are measuring how many votes each candidate will have. For that both methods must get intervals and a confidence level.
IanCal 8y ago
Perhaps a good way of putting it would be that there's probably not an extremely solid line like that 20%. You would leap at a bet with 10000-1 odds, and never go for 2-1. Let's say you can make these bets repeatedly. Would you always bet on 5.01-1 odds and never on 4.99-1 odds? Is it that odd to think that your team has about a 20% chance to win?

airza 8y ago

What? Confidence intervals are used to assign a confidence to a measure of a larger population.

asfasgasg 8y ago

I've been confused by this in the past too, so let me clarify for you. Confidence intervals are for predictions of a given value. They are not for probabilities. It doesn't make sense to say, "I think she has a 70% chance of winning, plus or minus 5%." On the other hand, it does make sense to say, I think she will get 48% of the vote, with an interval of +-1%.

scythe 8y ago

"Speaking of unknown probabilities or of probability of a probability must be forbidden as meaningless. " - Bruno de Finetti, 1977