3 ms·
I'm extremely rusty on my statistics, and I upvoted this because I find A/B tests interesting, but... are these numbers statistically significant? For the popu
by mcantor 17y ago
I'm extremely rusty on my statistics, and I upvoted this because I find A/B tests interesting, but... are these numbers statistically significant? For the population of internet users, are they actually practically significant? It just seems like the sample sizes and differences aren't really big enough to draw solid conclusions from. It doesn't say how long each test lasted for, either--what if the second test was done during peak hours?
- TimothyFitz 17y agoThis is exactly why I don't read other people's A/B experiment results. I haven't seen a single A/B experiment that listed it's statistical significance (and how that was calculated). I fear that bad data is worse than no data at all.
- brown9-2 17y agoGoogle Website Optimizer will give you a plus minus rate for your estimated conversion rate, which helps you to at least estimate what the significance/confidence is. For example, one of the pages in one of my current tests shows up as "Est. Conversion Rate: 17.0% +/- 1.4%".