3 ms·
I'm afraid that's not quite right. A simple python simulation will show you that a variant with -5% (ie NEGATIVE) uplift will still give a positive results arou
by martingoodson 13y ago
I'm afraid that's not quite right. A simple python simulation will show you that a variant with -5% (ie NEGATIVE) uplift will still give a positive results around 10% of the time if you perform early stopping of the test.
- paraschopra 13y agoNo matter which method you adopt, you cannot eliminate false positives entirely. You merely decrease / control the proportion of them.
- martingoodson 13y agoTo remove all doubt, your interpretation of the statistics is incorrect. In particular this sentence is demonstrably false: "They are directionally correct, [...] the business will still do better implementing the variation (v/s not doing anything)."