9 ms·
A year is far longer than they would need just to achieve statistical significance. They've got over 160 million unique visitors per month [1]. Even showing the
by aggie 12y ago
A year is far longer than they would need just to achieve statistical significance. They've got over 160 million unique visitors per month [1]. Even showing the variants to only 1% of traffic you're working with over 50,000 visitors every day, enough to run large multivariate tests.
1 - http://www.statista.com/statistics/271450/monthly-unique-visitors-to-us-retail-websites/ http://www.statista.com/statistics/271450/monthly-unique-vis...
- InclinedPlane 12y agoMaybe for some things, but not necessarily for everything amazon cares about, such as performance and reliability. For both of those amazon doesn't really care about averages, they care about the 3 9's experience. If there's a 0.5% chance that page load will take longer than some small number of milliseconds then that's not good enough for amazon and they'll go back to the drawing board. Factor in bugs in the new layout being fixed during the A/B test (necessitating resetting statistics) and it's easy to see how it could take a year to fully roll out a big change given amazon's cautiousness.
- randomsearch 12y agoStatistical significance is not the same as significance in business terms. I can't imagine anyone would use such an approach when deciding how much data to collect.
- jnbiche 12y ago>Statistical significance is not the same as significance in business terms Depending on the level of significance, why not? If it's expensive or simply not possible to gather the full data set, sampling is absolutely a valid basis for business decisions. Why wouldn't it be?
- tosh 12y agoAmazon probably also has vested interest in how a change affects LTV which is way harder to understand than whether there was a purchase regression in a short time window.