3 ms·
The reason you don’t collect some data and then see what curves you can fit to it is exactly the problem you describe. “Eureka, a ninth-order polynomial functio
by sokoloff 6y ago
The reason you don’t collect some data and then see what curves you can fit to it is exactly the problem you describe. “Eureka, a ninth-order polynomial function fits the data quite closely!”
You can’t look at your A/B test results every morning and stop the test the moment one measurement “shows significance”. That’s not how it works.
- marcosdumay 6y agoThat's why the GP was taking about weak priors. You just don't run an A/B test for 6 months hoping that a new design increases sales, get results pointing that the new design reduces sales by 90%, and then go make another 6 months long test to make sure the sales indeed fell. Acknowledging the numbers on that single dimension are different from what you expected isn't the same as testing a bunch of different dimensions. And if your hypothesis does not allow for it, you completely failed at rational thinking.