3 ms·

I am not sure I follow. Are you saying that a 90% chance of not needing an umbrella is the same as a 26% chance? These are vastly different probabilities which

by oezi 5y ago

I am not sure I follow. Are you saying that a 90% chance of not needing an umbrella is the same as a 26% chance? These are vastly different probabilities which mean that you might be wrong 9 out 10 days, or right 3 out of 4 times. As another example: I bet most companies would die to turn up conversion rates up even 1 or 2 percent. When dealing in risk management where probabilities should start in 1 in 10 or 1 in 100, then I could agree that 90% risk of failure isn't to different from 26%.

time_to_smile 5y ago
> These are vastly different probabilities When it comes to certainty about an outcome (i.e. a hypothesis), they're not really that different. Probabilities are not linear, and start at 0.5. In log-odds terms were talking about 2.19 vs -1.04... not really a shocking amount of information. Sure in terms of gambling odds this might be useful but not for making any concrete statements about the world. To your point: > I bet most companies would die to turn up conversion rates up even 1 or 2 percent. This is talking about two very different things. I would also be happy if I had a known 2 percent increasing in my stock performance. We're talking about quantifying uncertainty (though these criticism hold even in a frequentist frame work). I certainly wouldn't "die for" 2% absolute increase in my confidence about something (unless I'm 98% confident already!) We're talking about our confidence in a hypothesis. At no company I've worked at would I be anywhere near comfortable stating that a 0.9 probability that A > B is enough to call an A/B test on. I'm going to be very skeptical of any analysis I'm doing if mode coefficients have anywhere near that probability of a sign error. If you have no other information, then sure if I am 90% confident that one choice is stronger than other I'll go with the limited information that I have. However anyone with a statistics background would view either of these numbers as being very uncertain. That's arguably why these numbers are so good for betting, you want there to be a lot of uncertainty when gambling or it's no fun. But as far as statistics is considered both 0.9 and 0.26 are essentially uninteresting numbers, that lead to very low confidence in the conclusion being drawn.

by oezi 5y ago

time_to_smile 5y ago

> These are vastly different probabilities When it comes to certainty about an outcome (i.e. a hypothesis), they're not really that different. Probabilities are not linear, and start at 0.5. In log-odds terms were talking about 2.19 vs -1.04... not really a shocking amount of information. Sure in terms of gambling odds this might be useful but not for making any concrete statements about the world. To your point: > I bet most companies would die to turn up conversion rates up even 1 or 2 percent. This is talking about two very different things. I would also be happy if I had a known 2 percent increasing in my stock performance. We're talking about quantifying uncertainty (though these criticism hold even in a frequentist frame work). I certainly wouldn't "die for" 2% absolute increase in my confidence about something (unless I'm 98% confident already!) We're talking about our confidence in a hypothesis. At no company I've worked at would I be anywhere near comfortable stating that a 0.9 probability that A > B is enough to call an A/B test on. I'm going to be very skeptical of any analysis I'm doing if mode coefficients have anywhere near that probability of a sign error. If you have no other information, then sure if I am 90% confident that one choice is stronger than other I'll go with the limited information that I have. However anyone with a statistics background would view either of these numbers as being very uncertain. That's arguably why these numbers are so good for betting, you want there to be a lot of uncertainty when gambling or it's no fun. But as far as statistics is considered both 0.9 and 0.26 are essentially uninteresting numbers, that lead to very low confidence in the conclusion being drawn.