3 ms·
It's worth noting that splittester.com calculates 95% confidence that those two (identical) ads will have different long term response rates.
by TimothyFitz 16y ago
It's worth noting that splittester.com calculates 95% confidence that those two (identical) ads will have different long term response rates.
- ellyagg 16y agoSure because 95% leaves 5% of room to cherry pick aberrations.
- reitzensteinm 16y agoWhich should be a pretty big wake up call to anyone not doing the math. 1 in 20 events happen all the time, yet naively reading the results makes it look like one is almost certainly better than the other.
- TimothyFitz 16y agoAgreed, this is such a common problem. Another really common manifestation is when you A/B test while looking at multiple variables. If you observe the results of 20 variables, you should expect one of them to have 95% confidence!