3 ms·

> no confidence that we can really know anything for sure one way or another Not only is this true from a general sense of "fog of war" and misinformation, but

> no confidence that we can really know anything for sure one way or another Not only is this true from a general sense of "fog of war" and misinformation, but even in a purely statistical sense, these numbers aren't compelling. As a stats person, both 0.9 and 0.26 are probabilities that basically reflect nothing more than a vague sense how the outcome will go. In a classical NHST this degree of confidence would not pass even the most lose definitions of "statistical significance". And in a more Bayesian setting, being 90% and 26% sure of rain both mean you'll bing an umbrella. Having a 90% chance of survival and a 26% chance of survival means are both too risky for almost all scenarios. But for whatever reason, when predicting things like outcomes of an election or outcomes of a war, people start applying `round` to their probabilities. At work if I see that I'm either 90% sure or 26%, I report that we still don't have enough information to say anything particularly interesting or to justify any major risks. So even if we had reliable information about the situation, these numbers aren't interesting. Since we don't, they're useless.

oezi 5y ago
I am not sure I follow. Are you saying that a 90% chance of not needing an umbrella is the same as a 26% chance? These are vastly different probabilities which mean that you might be wrong 9 out 10 days, or right 3 out of 4 times. As another example: I bet most companies would die to turn up conversion rates up even 1 or 2 percent. When dealing in risk management where probabilities should start in 1 in 10 or 1 in 100, then I could agree that 90% risk of failure isn't to different from 26%.
- time_to_smile 5y ago
  > These are vastly different probabilities When it comes to certainty about an outcome (i.e. a hypothesis), they're not really that different. Probabilities are not linear, and start at 0.5. In log-odds terms were talking about 2.19 vs -1.04... not really a shocking amount of information. Sure in terms of gambling odds this might be useful but not for making any concrete statements about the world. To your point: > I bet most companies would die to turn up conversion rates up even 1 or 2 percent. This is talking about two very different things. I would also be happy if I had a known 2 percent increasing in my stock performance. We're talking about quantifying uncertainty (though these criticism hold even in a frequentist frame work). I certainly wouldn't "die for" 2% absolute increase in my confidence about something (unless I'm 98% confident already!) We're talking about our confidence in a hypothesis. At no company I've worked at would I be anywhere near comfortable stating that a 0.9 probability that A > B is enough to call an A/B test on. I'm going to be very skeptical of any analysis I'm doing if mode coefficients have anywhere near that probability of a sign error. If you have no other information, then sure if I am 90% confident that one choice is stronger than other I'll go with the limited information that I have. However anyone with a statistics background would view either of these numbers as being very uncertain. That's arguably why these numbers are so good for betting, you want there to be a lot of uncertainty when gambling or it's no fun. But as far as statistics is considered both 0.9 and 0.26 are essentially uninteresting numbers, that lead to very low confidence in the conclusion being drawn.

oezi 5y ago

I am not sure I follow. Are you saying that a 90% chance of not needing an umbrella is the same as a 26% chance? These are vastly different probabilities which mean that you might be wrong 9 out 10 days, or right 3 out of 4 times. As another example: I bet most companies would die to turn up conversion rates up even 1 or 2 percent. When dealing in risk management where probabilities should start in 1 in 10 or 1 in 100, then I could agree that 90% risk of failure isn't to different from 26%.

time_to_smile 5y ago
> These are vastly different probabilities When it comes to certainty about an outcome (i.e. a hypothesis), they're not really that different. Probabilities are not linear, and start at 0.5. In log-odds terms were talking about 2.19 vs -1.04... not really a shocking amount of information. Sure in terms of gambling odds this might be useful but not for making any concrete statements about the world. To your point: > I bet most companies would die to turn up conversion rates up even 1 or 2 percent. This is talking about two very different things. I would also be happy if I had a known 2 percent increasing in my stock performance. We're talking about quantifying uncertainty (though these criticism hold even in a frequentist frame work). I certainly wouldn't "die for" 2% absolute increase in my confidence about something (unless I'm 98% confident already!) We're talking about our confidence in a hypothesis. At no company I've worked at would I be anywhere near comfortable stating that a 0.9 probability that A > B is enough to call an A/B test on. I'm going to be very skeptical of any analysis I'm doing if mode coefficients have anywhere near that probability of a sign error. If you have no other information, then sure if I am 90% confident that one choice is stronger than other I'll go with the limited information that I have. However anyone with a statistics background would view either of these numbers as being very uncertain. That's arguably why these numbers are so good for betting, you want there to be a lot of uncertainty when gambling or it's no fun. But as far as statistics is considered both 0.9 and 0.26 are essentially uninteresting numbers, that lead to very low confidence in the conclusion being drawn.