5 ms·
Having been guilty of "can we try another algorithm" which is only one remove from p-jacking, I think we need to encourage more negative results publication. We
by ggm 6y ago
Having been guilty of "can we try another algorithm" which is only one remove from p-jacking, I think we need to encourage more negative results publication. We need to incentivise and reward it.
There should be a way to get tenure on it, at least partially.
- inter_netuser 6y ago10x bonus for every disproven dogma.
- veddox 6y ago"Dogmas" are pretty much never disproven. (Technically, outside of mathematics, they can't be.) Science is a long process of lots of people looking at lots of data and coming up with different explanations to make sense of it all. Some of these explanations make more sense than others, but it's pretty much guaranteed that even the experts won't be able to agree on which ones. As new data and new generations of researchers emerge, the explanations evolve, consolidate, and sometimes are discarded again. In short: science is a mess of opinions, gradually moving along. If you want to get a feel for what that looks like in practice, have a look at this blog post about diverging opinions in ecology: https://dynamicecology.wordpress.com/2018/04/30/poll-results-heres-what-our-readers-think-about-some-of-the-most-controversial-ideas-in-ecology/ https://dynamicecology.wordpress.com/2018/04/30/poll-results... (Most of the hypotheses he explores are current "textbook knowledge"!)
- wombatpm 6y agoA new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die and a new generation grows up that is familiar with it. . . . An important scientific innovation rarely makes its way by gradually winning over and converting its opponents: it rarely happens that Saul becomes Paul. What does happen is that its opponents gradually die out, and that the growing generation is familiarized with the ideas from the beginning: another instance of the fact that the future lies with the youth. — Max Planck, Scientific autobiography, 1950, p. 33, 97
- fabian2k 6y agoThe problem is that good negative results are also a lot of effort to achieve. You need to figure out if you have a real negative result, and not just some error or mistake in your experiments. And spending that additional effort when the result by itself is not interesting might not be justified.
- roenxi 6y agoIt is too easy to churn out negative results though. If someone were to hit this at an incentives level, I think it might be fruitful to require that anyone getting tenure has to have had a major study replicated by an independent researcher. That'd have complicated ramifications and probably encourage a very insular community.
- abnry 6y agoThat would improve incentives for those trying to get tenure, but what are the incentives for the replicators? That seems to be the biggest problem. It costs money, time, and with the current incentive structure, reputation.
- mhh__ 6y agoI think some lower pressure medium for publication could work. Publishing every attempt that didn't work would be tedious, but a database of attempts, implementation, and results would allow others to not walk into the same bog