34 ms·
#2 is a slippery slope if you don't do it properly. You might look end up looking at lots of different slices of your data, and you might come to the conclusio
by clarle 2y ago
#2 is a slippery slope if you don't do it properly.
You might look end up looking at lots of different slices of your data, and you might come to the conclusion, "Oh, it looks like France is statistically significant negative on our new signup flow changes".
It's important to make sure you have a hypothesis for the given slice before you start the experiment and not just hunt for outliers after the fact, or otherwise you're just p-hacking [1].
[1]: https://en.wikipedia.org/wiki/p-hacking https://en.wikipedia.org/wiki/p-hacking
- deleted 2y ago[deleted]
- lnenad 2y agoI agree, as per example and point number one, if your goals was to increase conversions, you were successful. You can then go to the next step, slice the data up, and iterate on another change. If you fall into the box of over-analyzing you will probably find all sorts of irrelevant patterns.
- Dyac 2y agoFundamentally, you can't use the same data to both generate and validate/disprove a hypothesis. Srgmenting and data dredging is fine provided you run a new test with fresh data to validate if there is a causal relationship in any correlations found.