3 ms·
I can make a number of counter-arguments. 1. The volunteer nature of peer review is not a direct cause of bad peer review. In some sub-fields, peer reviews wor
by abdullahkhalids 6y ago
I can make a number of counter-arguments.
1. The volunteer nature of peer review is not a direct cause of bad peer review. In some sub-fields, peer reviews works very well at providing feedback or stopping bad papers from being published. In others, not so much. The difference is the norms of the community.
2. There are plenty of licensing bodies in other areas (medicine/plumbing/etc) where the reviewing body lets some of the chaff through with the wheat. Again, paying people can help but is not sufficient. What matters are the norms of the community - which take a lot of hard work to change/build.
3. Please note that the peer review process will always have errors (it's a noisy classification algorithm). The debate should be on the magnitude of acceptable Type I and II errors.
4. Asking papers to have 100% details of the experiment is impossible. It takes 20% effort to get to 80% details. And an additional 4x effort to get the last 20% of the details. A lot of details are often the psycho-motor skills of the grad student, and can't always be written down. I agree that experimentalists should include more details, and release their data, but 100% is too high a bar to reach.
> Then the entire community can choose to accept, reject, or ignore your work.
5. The Facebook architecture fails at separating true news from fake news, and a similar architecture for science will similarly be worse than the current system. Popularity contests rarely work at reliably selecting the best.
- praestigiare 6y ago4 is a strawman. The comment you are replying to did not say 100% details, it said "a methods section that allows a third party to fully recreate your experiment."
- abdullahkhalids 6y agoI took their meaning to be that the paper should have enough details be such that say a junior grad student could set up the same equipment and do the exact process described in the paper, and get pretty much the same results. I don't think this is possible. What is possible is the following. It's sufficient that the paper has the crucial details. And whoever tries to replicate will still have to apply some creativity, change some details, and learn some skills, and ultimately get slightly different results. In other words, enough detail to "fully recreate [any sufficiently complicated] experiment" is not possible to write down. Obviously, I am not providing any evidence for my position, besides my personal observations.
- deleted 6y ago[deleted]