3 ms·
The code has often already been peer reviewed. My take on this is, it would lead to more gatekeeping. More a hole reviewers rejecting papers because they don't
by thumbuddy 3y ago
The code has often already been peer reviewed. My take on this is, it would lead to more gatekeeping. More a hole reviewers rejecting papers because they don't like the way a comment is written, or because someone used a programming style they don't prefer.
If the code can't be reproduced 7/10 times it's because the author doesn't want you to be able to reproduce it so they can continue researching it. 1 out of 10 times it's because of an error, and the remainder is fraud.
It's also important to NOT require code be shared for some fields. I know of several companies who review papers for prominent journals in their field. They take results from papers they review put them in their software to sell and then reject the papers so competitors have no idea some new useful thing exists. It's not illegal to do this by the way, infact there's no reason not too with blind reviewers as far as typical businesses are concerned.
The crisis of reproducibility is the worst in biological sciences and becomes the best toward purer sciences. That's its own problem, I won't go too far chasing that down here... Scientific reporting is mostly flawed not because of "oh silly scientist you made an error" but because of the social factors driving it 9/10 times. People just won't admit it, because they'd lose their positions if they did...
- krageon 3y agoI would say it is fraud significantly more than 20% of the time if it is in a fashionable field (e.g. machine learning). This is a depressing realisation I came to after spending a significant amount of time (months? Perhaps more than a year) attempting to reproduce ML results from papers, to test for commercial viability. The work itself was incredibly cool and motivating (IMO it is the missing link between cutting edge science and the business world, and if I had gotten better results perhaps we could have picked researchers to sponsor), but the depressing reality was that the academic world (which I left because I was frustrated with how far from practical it was) is an island. People do things there, it does not have to be true or work and very frequently it's concerned with publishing lies for money. It reflects corporate life very closely, except it pretends to be something it is not.