4 ms·

So you think the data is only relevant to the hypothesis you decided it's relevant for before you collect it? "Have you seen the fire sir?" "I have only seen s

So you think the data is only relevant to the hypothesis you decided it's relevant for before you collect it? "Have you seen the fire sir?" "I have only seen smoke" "But have you seen the fire?" "It's irrelevant because I was only looking for smoke!". I mean I know they teach it in naive introductory experiment design in some soft science departments but c'mon it's neither smart nor correct. If you need a separate experiments for everything because otherwise you will run into "one of the 100 hypothesis is bound to be correct" problem then your priors are too weak, you're relying on getting lucky anyway and just end up discarding useful datapoints. If it's 5 out of 100 hypothesis fitting your data or 5 hypothesis out of 100 experiments passing "statistical significance" check doesn't matter at the end of the day. Both lead to shaky conclusions. The data being relevant depends on the experiment design (how it's collected). It doesn't depend on what you thought it's relevant for beforehand.

sokoloff 6y ago
The reason you don’t collect some data and then see what curves you can fit to it is exactly the problem you describe. “Eureka, a ninth-order polynomial function fits the data quite closely!” You can’t look at your A/B test results every morning and stop the test the moment one measurement “shows significance”. That’s not how it works.
- marcosdumay 6y ago
  That's why the GP was taking about weak priors. You just don't run an A/B test for 6 months hoping that a new design increases sales, get results pointing that the new design reduces sales by 90%, and then go make another 6 months long test to make sure the sales indeed fell. Acknowledging the numbers on that single dimension are different from what you expected isn't the same as testing a bunch of different dimensions. And if your hypothesis does not allow for it, you completely failed at rational thinking.
renewiltord 6y ago
Well, you can test multiple hypotheses at the same time. You just can't decide what the hypotheses are after you collect the data because you can't reasonably determine the correction factor.
maccard 6y ago
> So you think the data is only relevant to the hypothesis you decided it's relevant for before you collect it No, the hypothesis is only valid before the data is collected. > Have you seen the fire sir?" "I have only seen smoke" "But have you seen the fire?" "It's irrelevant because I was only looking for smoke!". It's actually more like: "Have you seen fire at x?" "There's smoke at Y" All that means is that something is true about Y, not that x isn't on fire.

sokoloff 6y ago

The reason you don’t collect some data and then see what curves you can fit to it is exactly the problem you describe. “Eureka, a ninth-order polynomial function fits the data quite closely!” You can’t look at your A/B test results every morning and stop the test the moment one measurement “shows significance”. That’s not how it works.

marcosdumay 6y ago
That's why the GP was taking about weak priors. You just don't run an A/B test for 6 months hoping that a new design increases sales, get results pointing that the new design reduces sales by 90%, and then go make another 6 months long test to make sure the sales indeed fell. Acknowledging the numbers on that single dimension are different from what you expected isn't the same as testing a bunch of different dimensions. And if your hypothesis does not allow for it, you completely failed at rational thinking.

renewiltord 6y ago

Well, you can test multiple hypotheses at the same time. You just can't decide what the hypotheses are after you collect the data because you can't reasonably determine the correction factor.

maccard 6y ago

> So you think the data is only relevant to the hypothesis you decided it's relevant for before you collect it No, the hypothesis is only valid before the data is collected. > Have you seen the fire sir?" "I have only seen smoke" "But have you seen the fire?" "It's irrelevant because I was only looking for smoke!". It's actually more like: "Have you seen fire at x?" "There's smoke at Y" All that means is that something is true about Y, not that x isn't on fire.