4 ms·

The first rule of experiment design is to state your hypothesis and controls upfront, and test only according to those. You don't get to update those parameters

The first rule of experiment design is to state your hypothesis and controls upfront, and test only according to those. You don't get to update those parameters after the test is complete. Failing to reject the null hypothesis but then saying 'if <alternative hypothesis picked with knowledge of result> is true then <result> would make sense' is not science. This claim is useless non-information after the results are known. This is a completely new claim that must be validated with a new study designed around it. Maybe this alternative hypothesis is true, maybe it isn't. Hindsight of the original experiment's data can never shed light on that. Trying to pass off a revisionist hypothesis with advance knowledge of results, is very much grasping at straws. Reduced to the absurd, its as if someone predicted a coin flip would land on heads, and then revised their reasoning after seeing it land on tails and told you that their method of predicting coin flips was right all along except for that little <sampling bias/whatever>. I hope this is clear and makes sense, because this is one of the fundamental pillars of the scientific method. You state a hypothesis, then you test, then you arrive at a conclusion. You don't start with a conclusion and then revise your hypothesis to match it after knowing the outcome of the test.

fenomas 7y ago
> Trying to pass off a revisionist hypothesis with advance knowledge of results I think you're reading too much into the "first they found X but then they found Y" narrative in the top comment of this thread. There's nothing like that in the actual paper - the authors start with one hypothesis, gather the data, do a bunch of regressions and whatnot, and report all the results. Then their overall conclusion is that the balance of the evidence supports their hypothesis, but they note all the caveats. There's no revisionism or changing of hypotheses or whatever, that I can see.
EForEndeavour 7y ago
But they didn't update their experimental parameters after examining the data and results. They reported the results of their original study, found an interesting pattern, revisited the raw data to explore why this pattern might arise, and discussed the implications. As you point out, the natural next step would be to design a followup study that does control for the effect of knowing if auditions are blinded upon applicant quality. They're allowed and even obligated to discuss this "useless non-information," as you inaccurately describe it, in their paper. That's science. That coin-flip prediction example seems incomplete to me. Here's how I would extend it to relate to the topic at hand: you ask a researcher to predict the distribution of heads and tails for a specific coin, which is secretly a trick coin with heads on both sides. The scientist trains a prediction algorithm on thousands of coin-flips performed on many randomly selected coins in the wild. Upon testing your coin, they find that 100% of the outcomes are heads, statistically very different from the training data (as confirmed by the appropriate statistical methods). You seem to advocate an uncritical reporting of this result, with no allowance for discussing possible explanations beyond the strict study flow of "predict 50% ± X% heads, flip coin N times, observe 100% heads, report result, end of story". A responsible scientist would discuss possible explanations for the anomalous result, including biased instrumentation, biased training data, biased sampling, uncontrolled variables, and so on. Uncritically collecting data, grinding through statistical tests, and reporting p-values/effect sizes/confidence intervals is not science. Researchers are fully expected to interpret their results in the greater context of their field, while of course maintaining statistical rigour, which is necessary but not sufficient to do good science.
- deleted 7y ago
  [deleted]

fenomas 7y ago

> Trying to pass off a revisionist hypothesis with advance knowledge of results I think you're reading too much into the "first they found X but then they found Y" narrative in the top comment of this thread. There's nothing like that in the actual paper - the authors start with one hypothesis, gather the data, do a bunch of regressions and whatnot, and report all the results. Then their overall conclusion is that the balance of the evidence supports their hypothesis, but they note all the caveats. There's no revisionism or changing of hypotheses or whatever, that I can see.

EForEndeavour 7y ago

But they didn't update their experimental parameters after examining the data and results. They reported the results of their original study, found an interesting pattern, revisited the raw data to explore why this pattern might arise, and discussed the implications. As you point out, the natural next step would be to design a followup study that does control for the effect of knowing if auditions are blinded upon applicant quality. They're allowed and even obligated to discuss this "useless non-information," as you inaccurately describe it, in their paper. That's science. That coin-flip prediction example seems incomplete to me. Here's how I would extend it to relate to the topic at hand: you ask a researcher to predict the distribution of heads and tails for a specific coin, which is secretly a trick coin with heads on both sides. The scientist trains a prediction algorithm on thousands of coin-flips performed on many randomly selected coins in the wild. Upon testing your coin, they find that 100% of the outcomes are heads, statistically very different from the training data (as confirmed by the appropriate statistical methods). You seem to advocate an uncritical reporting of this result, with no allowance for discussing possible explanations beyond the strict study flow of "predict 50% ± X% heads, flip coin N times, observe 100% heads, report result, end of story". A responsible scientist would discuss possible explanations for the anomalous result, including biased instrumentation, biased training data, biased sampling, uncontrolled variables, and so on. Uncritically collecting data, grinding through statistical tests, and reporting p-values/effect sizes/confidence intervals is not science. Researchers are fully expected to interpret their results in the greater context of their field, while of course maintaining statistical rigour, which is necessary but not sufficient to do good science.

deleted 7y ago
[deleted]