4 ms·

The standard that COMPare is aiming to achieve, as far as I understand it, is the following: 1. Outcomes that will be measured for clinical trials need to be r

The standard that COMPare is aiming to achieve, as far as I understand it, is the following: 1. Outcomes that will be measured for clinical trials need to be registered and public ahead of the trial (this is already achieved with recent regulation, as mentioned in the article) 2. If planned outcomes change, the change needs to be reported and explained, with new outcomes being again clearly stated. I really struggle to see how enforcing this will prevent productive research. This is basic scientific hygiene, not fantasy. No one is arguing that studies shouldn't be adjusted, just that the adjustments should be clearly stated. Completely changing the goals of a study after failing on your initial goals, and then claiming success without mentioning any of the failures along the way is poor science, because you're masking a lot of potentially significant findings along the way that others can benefit from as well. Not to mention that these standards have been agreed to by all the largest journals decades ago, with big PR claims of how this is needed etc., yet are never followed in practice. The responses of some journals (NEJM, JAMA that I recall) have been shameful, and contradictory to standards they claim to follow.

nonbel 11y ago
Misrepresenting your methods is just fraud, so you will get no argument from me there. However, think about the difference between A) setting a point prediction of your model as the null hypothesis vs B) "two groups are the same and samples were independent, etc" as the null hypothesis. In the first case there is a very small range of plausible outcomes consistent with the researcher's theory. In the second case there is a very large range (usually 50%). In the case of scenario A, messing up the experiment (either on purpose, or accident) makes it more difficult to claim evidence consistent with your model. In the case of scenario B, the same thing makes this easier. So researchers using scenario A are incentivized to be as careful and account for as many different sources of error as possible. Under scenario B, the incentive is the opposite, the sloppier the study the easier it is to get results consistent with the researcher's theory.

nonbel 11y ago

Misrepresenting your methods is just fraud, so you will get no argument from me there. However, think about the difference between A) setting a point prediction of your model as the null hypothesis vs B) "two groups are the same and samples were independent, etc" as the null hypothesis. In the first case there is a very small range of plausible outcomes consistent with the researcher's theory. In the second case there is a very large range (usually 50%). In the case of scenario A, messing up the experiment (either on purpose, or accident) makes it more difficult to claim evidence consistent with your model. In the case of scenario B, the same thing makes this easier. So researchers using scenario A are incentivized to be as careful and account for as many different sources of error as possible. Under scenario B, the incentive is the opposite, the sloppier the study the easier it is to get results consistent with the researcher's theory.