3 ms·
Warning: bad statistics abound in the article. If you can't spot them, let me explain a bit why statistics do not work that way. A _lot_ of data is necessary t
by sklivvz1971 7y ago
Warning: bad statistics abound in the article. If you can't spot them, let me explain a bit why statistics do not work that way.
A _lot_ of data is necessary to call a result if the effect is relatively small. If not enough samples are taken, there will still be a "formal" result, but the level of certainty of it will be low. Of course, depending on how accessible that page is, it might take a very long time to achieve significance.
How much certainty do we need in the result? P-values are generally considered not a great measure, but they are standard, and they are what they are using (if I understand correctly). This can give you an idea of the numbers we'd like to see for that value: https://xkcd.com/1478/ https://xkcd.com/1478/
In other words, only the first A/B test has conclusive results: accordion is worse than wizard by 30% with 0.01 p-value. Of course, you can see that there's a significant result _because the effect is large_. Note that this gives us an idea of the _resolution_ of their test system (i.e., how many As and Bs they have and how much effect they can see with their test). So we know they see an effect of about 30% change.
Now: all the other results are wrongly reported. The next four are mistakenly reported as "no statistically significant difference in performance." This is a false statement. Having high p-values means that any effect is below the resolution achievable with those numbers. In other words, the correct statement would be either "the experiment was inconclusive" or "the effect, if any, is less than 30% change". This is quite different from claiming "no change."
The last result has 28%, and it has a p-value of 0.05. I would _not_ accept it at face value. Such a p-value means the result has 1-in-20 chances of being due to a random fluctuation of data. There are six such experiments just in the post, so it's not unthinkable that the effect could be a fluke. More info on this here: https://www.xkcd.com/882/ https://www.xkcd.com/882/
All in all, the only result found is that _in their case_ the wizard was much more effective than the accordion. The rest is not supported by the facts presented and furthermore any generalization of the result outside their particular case is unwarranted.
- danaris 7y ago> The next four are mistakenly reported as "no statistically significant difference in performance." This is a false statement. Having high p-values means that any effect is below the resolution achievable with those numbers. No...that's what "no statistically significant difference" means. In these kinds of statistical tests, literally all you can look for is a statistically significant difference. Either you find one, or you don't. Once you know that you've found one, you can say what size it is—a difference of 0.1% is still plenty statistically significant if your p-value is <0.001. It's just not very practically significant. There are a lot of different reasons you might not find a statistically significant difference, and simply knowing the p-values cannot tell you which ones are true here. That's why a lot of journals (at least in Psychology, the field I have some knowledge about) have been pushing for reporting additional tests and measures.
- sklivvz1971 7y agoSure, but "forgetting" that the significance level is a 30% effect is at the least misleading. A real 25% increase or decrease would not have been statistically significant.
- danaris 7y agoI'm sorry; I don't follow. Could you clarify?
- sklivvz1971 7y agoIf the author's setup can only statistically determine effects of 30% or higher, saying "there's no significant change" means "there's no significant change of 30% or more". Nothing can be said of changes of less than 30%, and that's a hell of a lot of uncertainty. One of the variants could give you 25% more results and yet, those experiments would not see such a result. As such, it is clear that the experiment is simply not thorough enough to imply the two alternatives are equivalent. Not in the least.
- danaris 7y agoWhere do you get that the setup can only determine effects of 30% or higher? I didn't see anything about that in the article, though I admit I only skimmed it. Simply saying that the statistically significant effect they found was 30% has absolutely no bearing on how large an effect they can detect...
- sklivvz1971 7y agoI can't know what is the resolution but neither does anyone else, which is part of the problem. They only show 2 results at 30% (p 0.01) and 25% (p 0.05) which makes me suspect they have a very low resolution. I can't prove it, but I don't need to. They claim results without providing convincing evidence, not me.