5 ms·
Andrew Gelman had an interesting 9-years-too-late idea for what to do in cases like Bem's article: https://statmodeling.stat.columbia.edu/2019/01/27/jpsp-done-
by chalst 5y ago
Andrew Gelman had an interesting 9-years-too-late idea for what to do in cases like Bem's article:
https://statmodeling.stat.columbia.edu/2019/01/27/jpsp-done-bem-paper-back-2010-click-find-surprisingly-simple-answer/ https://statmodeling.stat.columbia.edu/2019/01/27/jpsp-done-...
Rather than publish all of the dubious speculations and the analysis whose problems took time to emerge, simply publish the data. The advantage, in his words:
> In the example of the ESP study, if anything’s valuable it’s the data. Publishing the data would get the journal off the hook regarding fairness, open-mindedness, and not missing a scoop, while enabling others to move on reanalyses right away, and without saddling the journal with an embarrassing endorsement of a weak theory that, it turns out, was not really supported by data at all.
- MaxBarraclough 5y agoObvious question then: what are the arguments/forces opposing this? Just inertia in the publishing norms?
- Misdicorl 5y agoThe approach fails miserably when taken in aggregate. I would read five to ten papers from my own field a week while in grad school. If I had to do the analysis for myself as well there wouldn't be any time to do my own research. Add on twenty more papers a week from adjacent fields where I don't even know all the techniques and the magnitude of the problem hopefully becomes clearer.
- MaxBarraclough 5y agoThe raw figures would be available in addition to the paper itself, not instead of it.
- chalst 5y agoGelman's suggestion is how to handle articles with unlikely conclusions: don't publish the submitted article, though that presumably would be circulated as a preprint, but just the data. Then the journal is bringing attention to interesting data without endorsing the dubious analysis or conclusions.
- MaxBarraclough 5y agoThanks, must have read that too quickly :-P It doesn't strike me as as very practical step. Pre-registration seems much more sensible.
- csee 5y agoThere is an incentive against doing it. No researcher wants the embarrassment of having an error that they didn't know about discovered. Journals don't fancy the embarrassment or hassle either. It would serve broader scientific progress, but that's not who is deciding to hide the data.
- MaxBarraclough 5y agoA tragedy of the commons, then?
- teekert 5y agoIf you ever find yourself thinking like this, get the f out of science asap, you have no idea what you are doing.
- chestervonwinch 5y agoIt follows from your comment that we should have anonymous authors in addition to anonymous reviewers. Can a researcher establish a provable track record of doing quality work while remaining anonymous?
- ruined 5y agoonly if they are independently wealthy
- csee 5y agoThat would never happen, again because of incentives. Academics want to build their brand and publishing anonymously doesn't help with that.
- chestervonwinch 5y agoPresently, for sure. I was wondering in the hypothetical, too, though. Authors really just need to provably show that they have a strong influence/presence on the publication graph or something. Although, reducing a researcher's entire value to a single scalar index is probably fraught. Also, egos would never go for completely anonymous authorship system, I think.
- deleted 5y ago[deleted]
- chestervonwinch 5y agoThere's likely multiple forces, but at least one must be money. Someone's gotta host the data, right? That falls on either the universities or the publishers.
- lifeisstillgood 5y agoIncentives: Spend five years doing difficult / expensive / dangerous research, gathering data that demonstrates X. The data set is vast, and you can get ten papers out of it, one every six months with some awesome collaboration and possibly tenure. Do you do that or publish all your data in the first paper and watch a dozen grad students publish your next nine papers in three weeks ? Incentives matter yes. Science and data should be free and transparent. But if we pay peanuts and expect the monkeys to appreciate the applause of publication, we need to change our incentive structure. The reasons AI / ML researchers don't mind publishing the data early is because they already get 500k salaries and equity.
- nuncanada 5y agoThen there is a huge problem with how we publish Science... Decades ago, sharing all these huge datasets would have been quite difficult if not uneconomical. But nowadays there is no reason left for not publishing the data and hopefully get academic rewards for that... I have seen huge amounts of data not being digitized and lost because of this ownership mentality...
- rscho 5y agoIt is not a 'mentality' issue. It's a systemic issue. Researchers need to eat too, and it's already difficult enough that few are willing to sacrifice much for the sake of morals.
- CamperBob2 5y agopublish all your data in the first paper and watch a dozen grad students publish your next nine papers in three weeks ? The way I would expect this to work is, the grad students give you credit for gathering the data, which allows you to share in the prestige associated with all of their projects. No?
- sdenton4 5y agoAlas, Kepler is better known than Brahe... Acknowledgements don't fix the incentive problem.