Peer review cannot catch if they performed the procedure correctly (e.g. in a lab), but it is there to check things like: - Validity of the experimental setup

Peer review cannot catch if they performed the procedure correctly (e.g. in a lab), but it is there to check things like: - Validity of the experimental setup - Validity of the statistical analysis methodology - Validity of the conclusions Take a look at the items in this comment:[1] > Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses All these can be caught with peer review. > together with an obsession for pursuing fashionable trends of dubious importance In contrast, this problem is intensified with peer review. > In their quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world. Or they retrofit hypotheses to fit their data. Not the role of peer review to catch these, but making your data and analyses scripts open will allow these to be caught by anyone who wishes to. The problem we have with current publishing is that I could use this technique to get faulty results, publish in a prestigious journal, get a 100 citations on my paper, and one of those citations will be from the person who looked at my data and saw obvious biases in my selection (which cannot be caught by a mere peer review). That one citation pointing out how wrong I am is lost in the noise. We need a way to highlight that citation - merely making a simple graph of connections via references doesn't get us there. [1] https://news.ycombinator.com/item?id=36140540 https://news.ycombinator.com/item?id=36140540