5 ms·

"Take 95% certainty compared to 99%. Because 95 is pretty close to 99, it feels like the difference should be minimal. In reality, there’s a gulf between those

"Take 95% certainty compared to 99%. Because 95 is pretty close to 99, it feels like the difference should be minimal. In reality, there’s a gulf between those two benchmarks: At 95% certainty, you have 19 people saying “yes” and 1 person saying “no.” At 99% certainty, you have 99 people saying “yes” and 1 person saying “no.” It feels like a difference of four people when, in reality, it’s a difference of 80. That’s a much bigger difference than we expect." I'm a data scientist, I feel like this is a really wildly confusing way to put this.

enraged_camel 7y ago
Reminds me of when I was shopping for a windbreaker. A brand had two models: one that blocked 98% of the wind and another that blocked 99%. To a casual observer, it wouldn’t be immediately obvious why the second one was significantly more expensive. After all, it’s just a 1% difference! (Of course, the answer is obvious: the second one was twice as effective at blocking the wind.)
- eximius 7y ago
  That is a questionable interpretation. It means the 98% one lets twice as much air in, but it doesn't block twice as much wind because the amount of ambient wind remains constant. To be more explicit, if there are 100 units of wind and one blocks 98 units, the other blocks 99 units. So one lets twice as much air in than the other, but I wouldn't consider that saying that one was 'twice as effective'. The relative improvement is (99-98)/98 ~ 1%.
  - enraged_camel 7y ago
    Your comment illustrates the point I'm making. If windbreaker A let's in 2 units of wind out of 100 and windbreaker B lets in 1 unit of wind out of 100, that means windbreaker B is twice as effective at blocking wind, because it lets in half as much wind as windbreaker A.
    - kilburn 7y ago
      I don't agree. It may mean "A is two times worse than B at blocking wind", but not "B is twice as good (effective) as A" at it.
      - eximius 7y ago
        This is the point I wanted to make. English is terribly imprecise about these things.
        around_here 7y ago
        But that’s not the point though. What is the _real_ difference between 98% and 99% in real world terms? How much colder will you feel? Will it dry out your skin faster? How much faster? It’s a difference against zero.
        enraged_camel 7y ago
        >>It’s a difference against zero. Exactly. This is why it is accurate to say that windbreaker B is twice as effective.
CoffeePython 7y ago
Yeah I had to read this over 3 or 4 times before I understood exactly what the author was saying. Very unintuitive way of putting it.
- oogali 7y ago
  Can either of you kindly help us understand in layman’s terms? (Or point to an informative URL resource?)
  - jbattle 7y ago
    The way it makes sense to me is to invert the number you are looking at. 95 & 99 are very 'close together' But if you look at 5 & 1 - the first number is 5 times the second. Huge relative difference
    - around_here 7y ago
      Sure, but the deal is that at very small numbers, a massive change in % difference doesn’t translate in to a massive difference. It’s _entirely_ dependent on context. The raw stats are meaningless.
  - deleted 7y ago
    [deleted]
  - mekoka 7y ago
    The author is muddying the water for effect, that's why his explanation is confusing. To understand what he means, you should ignore the results expressed as percentages and instead focus on 1:19 vs 1:99. Now, imagine a grid, where the unit on the x-axis is "No", vs "Yes" on the y-axis. Place both studies in terms of their respective No/Yes relationship. Study A: 1:19, 2:38, 3:47, ... Study B: 1:99, 2:198, 3:297, ... Can you picture the difference in their slopes?
- milesskorpen 7y ago
  I think that a lot of statistical statements are just unintuitive and are very difficult to explain in terms people grok quickly.
dyeje 7y ago
My statistics background is a class in college. This statement made sense to me on the first read.
ssivark 7y ago
For this and sibling comments, one way to think more intuitively is in terms of odds p(yes)/p(no). So going from 99% to 99.9% is 10 fold improvement in conversion odds. That's why, for eg. "many 9s" of uptime is very hard.
ChristianBundy 7y ago
Fractions simplify the hell out of this. - 95% is 1/20 - 99% is 1/100 This makes it intuitive that 95% has 5 times more people saying "no" than 99%. The wording made sense, but IMO using percentage adds more complexity than necessary.
ww520 7y ago
IANADS and it took a reread to parse it. It's a bit weird but it draws attention to the point. For others interested, basically at 95% confidence level, there's a 5% chance due to random events. At 99%, the randomness is limited to 1%, a 5X improvement, i.e. a huge 500% improvement not a tiny 4% improvement.
SNACKeR99 7y ago
As far as I can tell, he's mixing units (or scales?). Here's my version. At 95% certainty, you have 95 people saying "yes" and 5 people saying "no." At 99% certainty, you have 99 people saying "yes" and 1 person saying "no." While that is 5x improvement in uncertainity, it's also only 4 more people, a 4.21% improvement in overall "yes" count. I can't see how comparing an N in 20 number to an N in 100 number makes sense. IANAS.
cuchoi 7y ago
And that's not how to interpret the 95 and 99 thresholds. Depending on what proportion of "Version B" ideas are truly effective, your false positive rate (what the author is concerned about) will be very far away from 5% or 1%. At the 95 threshold (p-value of 0.05) you could have that >50% of what you identified as an "effective" version are not really effective (false positive). This is a good article to build the intuition around it: https://www.statisticsdonewrong.com/p-value.html https://www.statisticsdonewrong.com/p-value.html In the end it will also depend on how costly it is to have a false negative. The author hints that in this case is very costly.
the_watcher 7y ago
I felt the same. It's also misleading and implies the common misunderstanding of certainty levels being equivalent to "% chance that you're right".
chimi 7y ago
The problem isn't with the math, it's with the language, because we intuitively apply the same units to both four and eighty, when it's not four people and eighty people, it's four people and eighty percent, because 4/5 = 80%.

enraged_camel 7y ago

Reminds me of when I was shopping for a windbreaker. A brand had two models: one that blocked 98% of the wind and another that blocked 99%. To a casual observer, it wouldn’t be immediately obvious why the second one was significantly more expensive. After all, it’s just a 1% difference! (Of course, the answer is obvious: the second one was twice as effective at blocking the wind.)

eximius 7y ago
That is a questionable interpretation. It means the 98% one lets twice as much air in, but it doesn't block twice as much wind because the amount of ambient wind remains constant. To be more explicit, if there are 100 units of wind and one blocks 98 units, the other blocks 99 units. So one lets twice as much air in than the other, but I wouldn't consider that saying that one was 'twice as effective'. The relative improvement is (99-98)/98 ~ 1%.
- enraged_camel 7y ago
  Your comment illustrates the point I'm making. If windbreaker A let's in 2 units of wind out of 100 and windbreaker B lets in 1 unit of wind out of 100, that means windbreaker B is twice as effective at blocking wind, because it lets in half as much wind as windbreaker A.
  - kilburn 7y ago
    I don't agree. It may mean "A is two times worse than B at blocking wind", but not "B is twice as good (effective) as A" at it.
    - eximius 7y ago
      This is the point I wanted to make. English is terribly imprecise about these things.
      - around_here 7y ago
        But that’s not the point though. What is the _real_ difference between 98% and 99% in real world terms? How much colder will you feel? Will it dry out your skin faster? How much faster? It’s a difference against zero.
        enraged_camel 7y ago
        >>It’s a difference against zero. Exactly. This is why it is accurate to say that windbreaker B is twice as effective.

CoffeePython 7y ago

Yeah I had to read this over 3 or 4 times before I understood exactly what the author was saying. Very unintuitive way of putting it.

oogali 7y ago
Can either of you kindly help us understand in layman’s terms? (Or point to an informative URL resource?)
- jbattle 7y ago
  The way it makes sense to me is to invert the number you are looking at. 95 & 99 are very 'close together' But if you look at 5 & 1 - the first number is 5 times the second. Huge relative difference
  - around_here 7y ago
    Sure, but the deal is that at very small numbers, a massive change in % difference doesn’t translate in to a massive difference. It’s _entirely_ dependent on context. The raw stats are meaningless.
- deleted 7y ago
  [deleted]
- mekoka 7y ago
  The author is muddying the water for effect, that's why his explanation is confusing. To understand what he means, you should ignore the results expressed as percentages and instead focus on 1:19 vs 1:99. Now, imagine a grid, where the unit on the x-axis is "No", vs "Yes" on the y-axis. Place both studies in terms of their respective No/Yes relationship. Study A: 1:19, 2:38, 3:47, ... Study B: 1:99, 2:198, 3:297, ... Can you picture the difference in their slopes?
milesskorpen 7y ago
I think that a lot of statistical statements are just unintuitive and are very difficult to explain in terms people grok quickly.

dyeje 7y ago

My statistics background is a class in college. This statement made sense to me on the first read.

ssivark 7y ago

For this and sibling comments, one way to think more intuitively is in terms of odds p(yes)/p(no). So going from 99% to 99.9% is 10 fold improvement in conversion odds. That's why, for eg. "many 9s" of uptime is very hard.

ChristianBundy 7y ago

Fractions simplify the hell out of this. - 95% is 1/20 - 99% is 1/100 This makes it intuitive that 95% has 5 times more people saying "no" than 99%. The wording made sense, but IMO using percentage adds more complexity than necessary.

ww520 7y ago

IANADS and it took a reread to parse it. It's a bit weird but it draws attention to the point. For others interested, basically at 95% confidence level, there's a 5% chance due to random events. At 99%, the randomness is limited to 1%, a 5X improvement, i.e. a huge 500% improvement not a tiny 4% improvement.

SNACKeR99 7y ago

As far as I can tell, he's mixing units (or scales?). Here's my version. At 95% certainty, you have 95 people saying "yes" and 5 people saying "no." At 99% certainty, you have 99 people saying "yes" and 1 person saying "no." While that is 5x improvement in uncertainity, it's also only 4 more people, a 4.21% improvement in overall "yes" count. I can't see how comparing an N in 20 number to an N in 100 number makes sense. IANAS.

cuchoi 7y ago

And that's not how to interpret the 95 and 99 thresholds. Depending on what proportion of "Version B" ideas are truly effective, your false positive rate (what the author is concerned about) will be very far away from 5% or 1%. At the 95 threshold (p-value of 0.05) you could have that >50% of what you identified as an "effective" version are not really effective (false positive). This is a good article to build the intuition around it: https://www.statisticsdonewrong.com/p-value.html https://www.statisticsdonewrong.com/p-value.html In the end it will also depend on how costly it is to have a false negative. The author hints that in this case is very costly.

the_watcher 7y ago

I felt the same. It's also misleading and implies the common misunderstanding of certainty levels being equivalent to "% chance that you're right".

chimi 7y ago

The problem isn't with the math, it's with the language, because we intuitively apply the same units to both four and eighty, when it's not four people and eighty people, it's four people and eighty percent, because 4/5 = 80%.