3 ms·
I'm not sure the author really understands the math (from his comments) but he's right that it's significant, where significant means "I am at least 95% confide
by NationalPark 6y ago
I'm not sure the author really understands the math (from his comments) but he's right that it's significant, where significant means "I am at least 95% confident that the observed improvement is caused by this change". The test is fine (assuming he didn't just check the results every day until he found one that would make for a fun blog post).
The reason it feels intuitively broken is because the conversion rate is so low. But there were about 10k impressions in his test.
- canjobear 6y agoThat’s not what significance means.
- NationalPark 6y agoI mean... yes it is? 95% confidence in an effect is what most people mean when they say "statistically significant".
- jonex 6y agoI think the objection is about the phrasing that can be interpreted as "I will only be wrong in at most 5% of significant results", while what we mean is "if there is no effect, we will only get so large differences in at most 5% of cases". It's not uncommon that people fail to notice the nuance here and it's quite important for correctly interpreting reults.
- akiselev 6y ago> The reason it feels intuitively broken is because the conversion rate is so low. But there were about 10k impressions in his test. Can't be 95% confident in something if natural variability drowns out the effect. That kind of conversion rate is one bot RNG away from returning to the mean (noise), it just takes some domain knowledge to recognize why p-hacking is an illusion.