3 ms·
I really love how this very clearly visualises the need for statistical significance. To me, a novice, the A/A/B chart is wildly more illustrative of a point th
by skrebbel 12y ago
I really love how this very clearly visualises the need for statistical significance. To me, a novice, the A/A/B chart is wildly more illustrative of a point than the an A/B chart based of the same sample data with some significance number next to it. I understand from some of the comments here that there's all kinds of ways in which this A/A/B thing is subpar. But, if the chance of someone misinterpreting a chart decreases more than the chance of the chart itself being misleading, then isn't it a big win?
I'm really nerd sniped here. Is there any branch of statistics that focuses on human understanding? For example, there's all kinds of blogs and stories out there about how doctors routinely make wrong choices because they don't understand statistics well enough. Is there any serious body of knowledge that explores ways of getting these doctors to make these mistakes less frequently, without having to send them to sites with titles like "An Intuitive Explanation of Eliezer Yudkowsky’s Intuitive Explanation of Bayes’ Theorem"?
- marvy 12y agoI have no idea. But I will chime in with one factoid I've heard a few times. If you say things like "the false positive rate is 9%", people's intuitions lead them astray, but if you say things like "9/100 healthy people are labeled as sick by this test", then intuitions work much better.
- dustintran 12y agoThe problem is the more than likely chance that only doing one resample (in bootstrap terminology) may not lead to any clear rejection of statistical significance. That is, it is quite likely that in practice of this "A/A/B" testing, the second control group may look pretty much like the first, and so you can't clearly argue that the control deviates far away or not from the "Great Idea". What do you then? Well, the natural answer is to try the resample again. Do this n times, get an average for the variability, and that is precisely bootstrap.
- autokad 12y agobusiness and making money is always about a trade off of more information and making quicker decisions cheaply. the nice thing about business/making money is that detail doesn't matter, small information gain on your decisions can pay huge dividends. i mean the data is usually dirty and discusting in the first place