3 ms·
Importantly, to get the data to learn from it to improve faster than their competitors can.
by onlyrealcuzzo 5y ago
Importantly, to get the data to learn from it to improve faster than their competitors can.
- withinboredom 5y agoWe realized our competitors were copying us at a company long ago. We geofenced a/b tests so they wouldn’t see what we were testing and intentionally made the web app worse for that area (mostly as a “fun” 20% type project). Interestingly, the “worse” version starting doing really well even though we were giving them “control” for all older features. We ended up rolling back several months of changes to the entire product and we saw a huge burst in revenue. It wouldn’t be until years later until I understood the Simpson paradox and that a/b testing improves the mean of whatever metric you’re trying to improve, usually by decreasing some important segment and improving some unimportant segment.
- d0100 5y agoI would love to read a blog post with more details
- onlyrealcuzzo 5y agoThat's why you should measure many things - especially all your important things - in every a/b test [=
- withinboredom 5y agoWith the Simpson paradox, even if your metrics go up, across all your segments, it’s actually going down. Measuring only takes you so far.
- onlyrealcuzzo 5y agoSure - this is why Google Search constantly makes less money. Measuring only takes it so far.
- HWR_14 5y agoThat description of Simpson's Paradox doesn't really agree with my understanding. Simpson's paradox goes with how the same data is cut into subgroups and can invert the numbers. That is, it's less about "metrics up, other unmeasured metrics down" and more "we're up in all subgroups, so we never noticed that our numbers across when we combine the subgroups are down."