4 ms·

A position espoused by Bill Phillips [1], and to which I now adhere: "You should be willing to take either side of the bet that confidence interval implies." (

by ISL 3y ago

A position espoused by Bill Phillips [1], and to which I now adhere: "You should be willing to take either side of the bet that confidence interval implies." (paraphrasing; he says it better). For a concrete example, with a 95% confidence interval, you should be as willing to accept the 19:1 odds that the true value is outside the interval as you are the 1:19 odds that the true value is inside the interval. Aside from being generally correct, this approach is immediately actionable by making the meaning more visceral in discussions of uncertainty. Done right, it pushes you to assign uncertainties that are neither too conservative nor too optimistic. If the notion of letting your reader take either side of the bet makes your stomach a little queasy, you're on the right track. The feeling will subside when you're pretty sure you got the errorbar right and your reasoning is documented and defensible. Edit for OP's explicit question: One standard-deviation errorbars are 68% confidence intervals. Two standard deviations are 95% confidence intervals. (assuming you're a frequentist, of course) [1] https://www.nobelprize.org/prizes/physics/1997/phillips/facts/ https://www.nobelprize.org/prizes/physics/1997/phillips/fact...

eru 3y ago
> Edit for OP's explicit question: One standard-deviation errorbars are 68% confidence intervals. Two standard deviations are 95% confidence intervals. (assuming you're a frequentist, of course) Also assuming normal distribution, I think? > If the notion of letting your reader take either side of the bet makes your stomach a little queasy, you're on the right track. The feeling will subside when you're pretty sure you got the errorbar right and your reasoning is documented and defensible. > For a concrete example, with a 95% confidence interval, you should be as willing to accept the 19:1 odds that the true value is outside the interval as you are the 1:19 odds that the true value is inside the interval. I would like to build some edge into my bets. If a reader takes both sides of your example, they would be come out exactly even. But since readers are not forced to take any side at all, they will only take the bet if one of the sides has an advantage. So I would like to be able to say, 20:1 payout the true value is inside the error bar, and 1:10 payout it's outside the error bar (or something like that). The tighter the spread I am willing to quote, the more confident I am that I got the error estimates right. (I'm not sure how you translate these spreads back into the language of statistics.)
- jrumbut 3y ago
  > Also assuming normal distribution, I think? 95% is 95% regardless of the distribution. > I would like to build some edge into my bets. If a reader takes both sides of your example, they would be come out exactly even. You can imagine yourself being equally unhappy to take either side of the bet, if that's easier than imagining yourself being happy to take either side. It is for me, which is probably something to bring up in therapy. I also think that framing things as bets brings in all the cultural baggage around gambling and so it isn't always helpful. I'm not sure what a better framing is though.
  - eru 3y ago
    > 95% is 95% regardless of the distribution. Standard deviation away from the mean don't correspond to the same percentiles for all distributions, or do they? If you want to be (almost) independent of distribution, you need Chebyshev's inequality. But that one is far weaker. > Its practical usage is similar to the 68–95–99.7 rule, which applies only to normal distributions. Chebyshev's inequality is more general, stating that a minimum of just 75% of values must lie within two standard deviations of the mean and 88.89% within three standard deviations for a broad range of different probability distributions.[1][2] https://en.wikipedia.org/wiki/Chebyshev%27s_inequality https://en.wikipedia.org/wiki/Chebyshev%27s_inequality > I also think that framing things as bets brings in all the cultural baggage around gambling and so it isn't always helpful. I'm not sure what a better framing is though. Underwriting insurance without going bankrupt, perhaps?
    - jrumbut 3y ago
      > Standard deviation away from the mean don't correspond to the same percentiles for all distributions, or do they? If they had said a two standard deviation interval then you would have needed to know the distribution, but they said 95% which gives you all the information you need to make the bet.
      - eru 3y ago
        I was replying to this addendum, which only really works for the normal distribution: > Edit for OP's explicit question: One standard-deviation errorbars are 68% confidence intervals. Two standard deviations are 95% confidence intervals. (assuming you're a frequentist, of course) You are right about the first part of the original comment.
bluecheese452 3y ago
Expected value does not equal utility. I am not willing to mortgage my 1 million dollar house for a 1 in a 1000 shot at a billion.
- ISL 3y ago
  Then one should be very careful when assigning 99.9% confidence intervals.
  - bluecheese452 3y ago
    The two are unrelated.

by ISL 3y ago

eru 3y ago

> Edit for OP's explicit question: One standard-deviation errorbars are 68% confidence intervals. Two standard deviations are 95% confidence intervals. (assuming you're a frequentist, of course) Also assuming normal distribution, I think? > If the notion of letting your reader take either side of the bet makes your stomach a little queasy, you're on the right track. The feeling will subside when you're pretty sure you got the errorbar right and your reasoning is documented and defensible. > For a concrete example, with a 95% confidence interval, you should be as willing to accept the 19:1 odds that the true value is outside the interval as you are the 1:19 odds that the true value is inside the interval. I would like to build some edge into my bets. If a reader takes both sides of your example, they would be come out exactly even. But since readers are not forced to take any side at all, they will only take the bet if one of the sides has an advantage. So I would like to be able to say, 20:1 payout the true value is inside the error bar, and 1:10 payout it's outside the error bar (or something like that). The tighter the spread I am willing to quote, the more confident I am that I got the error estimates right. (I'm not sure how you translate these spreads back into the language of statistics.)

jrumbut 3y ago
> Also assuming normal distribution, I think? 95% is 95% regardless of the distribution. > I would like to build some edge into my bets. If a reader takes both sides of your example, they would be come out exactly even. You can imagine yourself being equally unhappy to take either side of the bet, if that's easier than imagining yourself being happy to take either side. It is for me, which is probably something to bring up in therapy. I also think that framing things as bets brings in all the cultural baggage around gambling and so it isn't always helpful. I'm not sure what a better framing is though.
- eru 3y ago
  > 95% is 95% regardless of the distribution. Standard deviation away from the mean don't correspond to the same percentiles for all distributions, or do they? If you want to be (almost) independent of distribution, you need Chebyshev's inequality. But that one is far weaker. > Its practical usage is similar to the 68–95–99.7 rule, which applies only to normal distributions. Chebyshev's inequality is more general, stating that a minimum of just 75% of values must lie within two standard deviations of the mean and 88.89% within three standard deviations for a broad range of different probability distributions.[1][2] https://en.wikipedia.org/wiki/Chebyshev%27s_inequality https://en.wikipedia.org/wiki/Chebyshev%27s_inequality > I also think that framing things as bets brings in all the cultural baggage around gambling and so it isn't always helpful. I'm not sure what a better framing is though. Underwriting insurance without going bankrupt, perhaps?
  - jrumbut 3y ago
    > Standard deviation away from the mean don't correspond to the same percentiles for all distributions, or do they? If they had said a two standard deviation interval then you would have needed to know the distribution, but they said 95% which gives you all the information you need to make the bet.
    - eru 3y ago
      I was replying to this addendum, which only really works for the normal distribution: > Edit for OP's explicit question: One standard-deviation errorbars are 68% confidence intervals. Two standard deviations are 95% confidence intervals. (assuming you're a frequentist, of course) You are right about the first part of the original comment.

bluecheese452 3y ago

Expected value does not equal utility. I am not willing to mortgage my 1 million dollar house for a 1 in a 1000 shot at a billion.

ISL 3y ago
Then one should be very careful when assigning 99.9% confidence intervals.
- bluecheese452 3y ago
  The two are unrelated.