7 ms·

I think one of the problems is that it often doesn't feel like cheating. You have a hypothesis, but you don't preregister it. It lives only in your head. You do

I think one of the problems is that it often doesn't feel like cheating. You have a hypothesis, but you don't preregister it. It lives only in your head. You do an experiment, you look at the results. They fail to confirm the hypothesis. But if you tweak the hypothesis just a little, the data suddenly confirm it. So why not publish the tweaked version? This is a malpractice that effectively invalidates the research. But it doesn't feel so. It feels more like a thought-crime.

AllegedAlec 8y ago
> You do an experiment, you look at the results. They fail to confirm the hypothesis. But if you tweak the hypothesis just a little, the data suddenly confirm it. So why not publish the tweaked version? If between tweaking the hypothesis and publishing it, you add the step "you perform another experiment which tests the tweaked hypothesis", you have just described the scientific method.
- raverbashing 8y ago
  I'm sure the budget for this new experiment will be given straight away, no questions asked. Though of course, there's a difference between "drug A doesn't work for condition B but seems to work slightly for C" and "drug A doesn't work for condition B but 10 of 12 individuals with condition C have shown significant improvement"
  - AllegedAlec 8y ago
    > I'm sure the budget for this new experiment will be given straight away, no questions asked. Most of the cases: yes, although your example of clinical trials is slightly different, and in that case, I do think the data should be publicly available to other researchers even in the case of a null.
  - xamuel 8y ago
    If the original experiment was large enough [1], you could almost always find some C such that "drug A doesn't work for condition B but 90% [2] of individuals with condition C showed significant improvement". In fact, you could replace [2] by a number arbitrarily close to 100% by increasing [1] accordingly.
    - wpietri 8y ago
      If the original experiment was large enough to do that, then somebody was given way too much money for the original experiment. So I'd expect that's a very rare case.
stkni 8y ago
>> But if you tweak the hypothesis just a little, the data suddenly confirm it This is 'data mining' right? And I've occasionally wondered about this, since I don't work in a scientific field but did once make use of the scientific method for some research I did. And yes the findings weren't especially conclusive but I'm not sure I could've tweaked the hypothesis to make it work. So, had I found something really interesting that didn't fit the hypothesis, is the 'right way' to conduct a new experiment from scratch? So say I did that, and used the 'tweaked' hypothesis, of course I'd find something interesting, because it's already there. In this new 'pre-registration' framework, how can I correct the problem and pursue the interesting idea but keep the science in-tact? Because, if I used some sort of cross-validation at the outset and I have all the data available I presumably can't change the sample, so the hypothesis presumably has to change.
- probably_wrong 8y ago
  Refining an experiment is not wrong. What is wrong, like you say, is going on a fishing expedition until you find a result you like. There are methods to account for follow-up experiments. Bonfaroni correction [1], for instance, requires you to increase your significance level with each new test. [1] https://en.m.wikipedia.org/wiki/Bonferroni_correction https://en.m.wikipedia.org/wiki/Bonferroni_correction
  - rumcajz 8y ago
    You can also split the dataset into two parts. Use first part to form a hypothesis. Register it. Then use the second part to confirm/disprove it.
  - soVeryTired 8y ago
    It's harder than you might think to control for multiple comparisons. The Bonfaroni correction assumes that each experiment is independent, and so penalises correlated experiments unnecessarily harshly. On the other hand, other tests typically require the researcher to make explicit assumptions on the correlation structure of the experiments despite the fact that it is not directly observable.
marcosdumay 8y ago
It can be much more sneaky than this. You have an hypothesis, do an experiment, it fails. You mark the hypothesis false and move on, never putting the work into publishing it (why would you?). At the same time, 19 other researchers have the same idea. Some 18 of them do the same as you, but one does the experiment and get a success. He will publish his work (why wouldn't he?), and it will be the only piece of literature available about the subject. Where on this narrative did anybody do anything even remotely unethical?

AllegedAlec 8y ago

> You do an experiment, you look at the results. They fail to confirm the hypothesis. But if you tweak the hypothesis just a little, the data suddenly confirm it. So why not publish the tweaked version? If between tweaking the hypothesis and publishing it, you add the step "you perform another experiment which tests the tweaked hypothesis", you have just described the scientific method.

raverbashing 8y ago
I'm sure the budget for this new experiment will be given straight away, no questions asked. Though of course, there's a difference between "drug A doesn't work for condition B but seems to work slightly for C" and "drug A doesn't work for condition B but 10 of 12 individuals with condition C have shown significant improvement"
- AllegedAlec 8y ago
  > I'm sure the budget for this new experiment will be given straight away, no questions asked. Most of the cases: yes, although your example of clinical trials is slightly different, and in that case, I do think the data should be publicly available to other researchers even in the case of a null.
- xamuel 8y ago
  If the original experiment was large enough [1], you could almost always find some C such that "drug A doesn't work for condition B but 90% [2] of individuals with condition C showed significant improvement". In fact, you could replace [2] by a number arbitrarily close to 100% by increasing [1] accordingly.
  - wpietri 8y ago
    If the original experiment was large enough to do that, then somebody was given way too much money for the original experiment. So I'd expect that's a very rare case.

stkni 8y ago

>> But if you tweak the hypothesis just a little, the data suddenly confirm it This is 'data mining' right? And I've occasionally wondered about this, since I don't work in a scientific field but did once make use of the scientific method for some research I did. And yes the findings weren't especially conclusive but I'm not sure I could've tweaked the hypothesis to make it work. So, had I found something really interesting that didn't fit the hypothesis, is the 'right way' to conduct a new experiment from scratch? So say I did that, and used the 'tweaked' hypothesis, of course I'd find something interesting, because it's already there. In this new 'pre-registration' framework, how can I correct the problem and pursue the interesting idea but keep the science in-tact? Because, if I used some sort of cross-validation at the outset and I have all the data available I presumably can't change the sample, so the hypothesis presumably has to change.

probably_wrong 8y ago
Refining an experiment is not wrong. What is wrong, like you say, is going on a fishing expedition until you find a result you like. There are methods to account for follow-up experiments. Bonfaroni correction [1], for instance, requires you to increase your significance level with each new test. [1] https://en.m.wikipedia.org/wiki/Bonferroni_correction https://en.m.wikipedia.org/wiki/Bonferroni_correction
- rumcajz 8y ago
  You can also split the dataset into two parts. Use first part to form a hypothesis. Register it. Then use the second part to confirm/disprove it.
- soVeryTired 8y ago
  It's harder than you might think to control for multiple comparisons. The Bonfaroni correction assumes that each experiment is independent, and so penalises correlated experiments unnecessarily harshly. On the other hand, other tests typically require the researcher to make explicit assumptions on the correlation structure of the experiments despite the fact that it is not directly observable.

marcosdumay 8y ago

It can be much more sneaky than this. You have an hypothesis, do an experiment, it fails. You mark the hypothesis false and move on, never putting the work into publishing it (why would you?). At the same time, 19 other researchers have the same idea. Some 18 of them do the same as you, but one does the experiment and get a success. He will publish his work (why wouldn't he?), and it will be the only piece of literature available about the subject. Where on this narrative did anybody do anything even remotely unethical?