15 ms·

I didn't understand this part. Why is the author changing scales? Shouldn't it be that at: At 95% certainty, you have 95 people saying "yes" and 5 people sayin

I didn't understand this part. Why is the author changing scales? Shouldn't it be that at: At 95% certainty, you have 95 people saying "yes" and 5 people saying "no"? So it's easier to make an apples to apples comparison? What point is the author trying to make with changing the scale?

tempguy9999 7y ago
Though obviously equivalent, I think as a percentage 95/5 and as a reduced fraction 19/1, they both have value in different ways to the non-mathematician. Giving 2 intuitive handles on a thing instead of one is helpful.
meigwilym 7y ago
Especially as he goes on to state: > It feels like a difference of four people when, in reality, it’s a difference of 80. That’s a much bigger difference than we expect. I had to stop reading here.
- Marazan 7y ago
  But that's the point. Ratios are really hard to reason about. Your mind sees the 5% going to 1% (or the 95% going to 99%) and thinks it's a small difference. When in actuality it's a big change.
- UserIsUnused 7y ago
  Depends on how you frame it. There is 80 of difference when you think about "how many users say yes for each No" So, what do you want to know, "yes for every 100" or "yes for every no" ? It matters.
- Ensorceled 7y ago
  When you thought about it some more, did you come back?
arnvald 7y ago
The point is to show that with 95% for every 1 person saying "no" you have 19 people saying "yes". With 99% for every 1 person saying "no" you have 99 people saying "yes". The common denominator is one "no".
- ashelmire 7y ago
  That’s not how this works though. The same number of people are landing on your site. You aren’t getting 80 more signups as suggested.
  - kryogen1c 7y ago
    > That’s not how this works though. What youre struggling with is the counterintuitive nature of applied statistics vs pure math, and this is the point TFA was trying to make. > You aren’t getting 80 more signups as suggested TFA isnt saying you "get" more, but just illustrating how different 95 and 99% actually are. Its restating the potato paradox, linked elsewhere in the thread https://en.m.wikipedia.org/wiki/Potato_paradox https://en.m.wikipedia.org/wiki/Potato_paradox
    - bobwaycott 7y ago
      Genuinely curious aside—you used TFA twice. It means “the fucking article” here, right? When you use TFA, your otherwise helpful comment reads as if it’s angry, exasperated, or some otherwise negative feeling that stands in opposition to the rest of your comment—which reads as a genuine attempt to be helpful and explain. Why do use TFA?
      - lixtra 7y ago
        TFA can also stand for The Fine Article. By HN rules you should assume the best; in this case for you, he was talking about the fine article.
        bobwaycott 7y ago
samastur 7y ago
I'm not the author of the article, but I sometimes use the same approach because in my experience saying 5 people out of 100 does not have the same effect as 1 in 20 even though mathematically they are same. It provides a mental check if you are still ok with your original perception.
dkersten 7y ago
Comparing A:B against C:D is much less intuitive for us puny humans than comparing A:B against C:B.
mytailorisrich 7y ago
He is not changing scale. He is just illustrating what the probability means in terms of the size of the set you need to expect the event to occur once, which has a more practical value than a probability.
Ensorceled 7y ago
> So it's easier to make an apples to apples comparison? That's exactly the point. How many people said yes for every one who said no: 50% -> 1 yes for every no 95% -> 19 yeses for every no 99% -> 99 yeses for every no 99.9% -> 999 yeses for every no
bigiain 7y ago
In the context of A/B Testing, if you've decided to stop the test at "95% significance" then you'll stop at the 19 yes, 1 no spot (or maybe 38:2). If you're testing for 99% significance, you won't stop unless you get to 99:1 (or more likely, you'll need to test way past that because "no" number 2 will arrive before "yes" number 99, so you'll need to test to 198:2 or 297:3 or further. This is why you should never stop an A/B test once you've "hit your statistical significance". Always choose the number of tests you'd need to prove the significance before you start, and let it run even if it's "obviously winning" (or losing).
- krumpet 7y ago
  Can you elaborate on this? When an A/B test has hit the desired significance, what is the value obtained in testing further?

tempguy9999 7y ago

Though obviously equivalent, I think as a percentage 95/5 and as a reduced fraction 19/1, they both have value in different ways to the non-mathematician. Giving 2 intuitive handles on a thing instead of one is helpful.

meigwilym 7y ago

Especially as he goes on to state: > It feels like a difference of four people when, in reality, it’s a difference of 80. That’s a much bigger difference than we expect. I had to stop reading here.

Marazan 7y ago
But that's the point. Ratios are really hard to reason about. Your mind sees the 5% going to 1% (or the 95% going to 99%) and thinks it's a small difference. When in actuality it's a big change.
UserIsUnused 7y ago
Depends on how you frame it. There is 80 of difference when you think about "how many users say yes for each No" So, what do you want to know, "yes for every 100" or "yes for every no" ? It matters.
Ensorceled 7y ago
When you thought about it some more, did you come back?

arnvald 7y ago

The point is to show that with 95% for every 1 person saying "no" you have 19 people saying "yes". With 99% for every 1 person saying "no" you have 99 people saying "yes". The common denominator is one "no".

ashelmire 7y ago
That’s not how this works though. The same number of people are landing on your site. You aren’t getting 80 more signups as suggested.
- kryogen1c 7y ago
  > That’s not how this works though. What youre struggling with is the counterintuitive nature of applied statistics vs pure math, and this is the point TFA was trying to make. > You aren’t getting 80 more signups as suggested TFA isnt saying you "get" more, but just illustrating how different 95 and 99% actually are. Its restating the potato paradox, linked elsewhere in the thread https://en.m.wikipedia.org/wiki/Potato_paradox https://en.m.wikipedia.org/wiki/Potato_paradox
  - bobwaycott 7y ago
    Genuinely curious aside—you used TFA twice. It means “the fucking article” here, right? When you use TFA, your otherwise helpful comment reads as if it’s angry, exasperated, or some otherwise negative feeling that stands in opposition to the rest of your comment—which reads as a genuine attempt to be helpful and explain. Why do use TFA?
    - lixtra 7y ago
      TFA can also stand for The Fine Article. By HN rules you should assume the best; in this case for you, he was talking about the fine article.
      - bobwaycott 7y ago

samastur 7y ago

I'm not the author of the article, but I sometimes use the same approach because in my experience saying 5 people out of 100 does not have the same effect as 1 in 20 even though mathematically they are same. It provides a mental check if you are still ok with your original perception.

dkersten 7y ago

Comparing A:B against C:D is much less intuitive for us puny humans than comparing A:B against C:B.

mytailorisrich 7y ago

He is not changing scale. He is just illustrating what the probability means in terms of the size of the set you need to expect the event to occur once, which has a more practical value than a probability.

Ensorceled 7y ago

> So it's easier to make an apples to apples comparison? That's exactly the point. How many people said yes for every one who said no: 50% -> 1 yes for every no 95% -> 19 yeses for every no 99% -> 99 yeses for every no 99.9% -> 999 yeses for every no

bigiain 7y ago

In the context of A/B Testing, if you've decided to stop the test at "95% significance" then you'll stop at the 19 yes, 1 no spot (or maybe 38:2). If you're testing for 99% significance, you won't stop unless you get to 99:1 (or more likely, you'll need to test way past that because "no" number 2 will arrive before "yes" number 99, so you'll need to test to 198:2 or 297:3 or further. This is why you should never stop an A/B test once you've "hit your statistical significance". Always choose the number of tests you'd need to prove the significance before you start, and let it run even if it's "obviously winning" (or losing).

krumpet 7y ago
Can you elaborate on this? When an A/B test has hit the desired significance, what is the value obtained in testing further?