4 ms·

If you put ethics aside, then yes there was value. Failed research is most valuable, while successful research is usually quite worthless due to the bias toward

If you put ethics aside, then yes there was value. Failed research is most valuable, while successful research is usually quite worthless due to the bias toward finding whatever researchers already believe. I do wonder if they would have published if they didn't expect disclosure by angry Linux maintainers, i.e. if they really believe they were weakly successful. Generally, I think this is the type of finding that normally gets lost if there's no pre-disclosure. https://pubmed.ncbi.nlm.nih.gov/16060722/ https://pubmed.ncbi.nlm.nih.gov/16060722/

ludamad 5y ago
The ideas in the paper were novel, perhaps (didn't check) but there was far more to learn from looking at reviews where bugs did slip by than doing their experiment. You could probably calculate the bug acceptance rate by category of bug, making a few random data points does not help science here
- foolmeonce 5y ago
  I think the prevailing belief, and belief they must have had to try their research was the belief that a good percentage of attempts would slip through initial review to be caught at a later stage. I fail to see what your alternative research would do to that apparently false belief other than reinforce it.
webmaven 5y ago
> If you put ethics aside, then yes there was value. Failed research is most valuable, while successful research is usually quite worthless due to the bias toward finding whatever researchers already believe. True as far as it goes, but the most valuable failing research is that which fails to produce an expected positive result. Next most valuable is failing to corroborate a novel hypothesis (which is probably what you meant). When you get an expected result, you haven't learned much, regardless of whether you were (or should have been) expecting success OR failure (with the exception being getting more, or more accurate, data that narrows error bars).

ludamad 5y ago

The ideas in the paper were novel, perhaps (didn't check) but there was far more to learn from looking at reviews where bugs did slip by than doing their experiment. You could probably calculate the bug acceptance rate by category of bug, making a few random data points does not help science here

foolmeonce 5y ago
I think the prevailing belief, and belief they must have had to try their research was the belief that a good percentage of attempts would slip through initial review to be caught at a later stage. I fail to see what your alternative research would do to that apparently false belief other than reinforce it.

webmaven 5y ago

> If you put ethics aside, then yes there was value. Failed research is most valuable, while successful research is usually quite worthless due to the bias toward finding whatever researchers already believe. True as far as it goes, but the most valuable failing research is that which fails to produce an expected positive result. Next most valuable is failing to corroborate a novel hypothesis (which is probably what you meant). When you get an expected result, you haven't learned much, regardless of whether you were (or should have been) expecting success OR failure (with the exception being getting more, or more accurate, data that narrows error bars).