3 ms·

(PhD student in STEM here) I think most people have the wrong idea about what peer review is. My advisor teaches us to treat it as a first check, but it is not

(PhD student in STEM here) I think most people have the wrong idea about what peer review is. My advisor teaches us to treat it as a first check, but it is not a guarantee of correct results. Most of the time I spend on research is actually trying to understand the literature and reproducing their results. If I can't do it, it probably means I don't understand enough about the work I'm reading, but there is also the small chance that the published analysis is wrong, which already happened to me. EDIT: typo.

gowld 3y ago
Most people have the wrond idea about what publication is, too. For almost all papers, for almost all readers, it doesn't matter if results are correct. It's just part of the game of academia. If you need to build something based on a paper, and you need it to work correctly (so, outside of public policy or macroeconomics), you need to do the work yourself. But if you just want to cite the work and write your own paper on top of it, go ahead. Published science is like ChatGPT ;-)
- slt2021 3y ago
  scientific publication is like a JIRA ticket. to get visibility of your work you need to close JIRA tickets, the more the better, since people look at aggregate metrics. The more your JIRA tickets are quoted in company documentation the better. but JIRA ticket is not code, in the same way publication is not science
caddemon 3y ago
It's never going to be a guarantee, but it's also implemented very shittily at present. There's a difference between what peer review is and what it should be. It makes sense to look at it realistically in day to day practice, but in discussion of systemic problems it should be viewed in a different light.
BeetleB 3y ago
Peer review cannot catch if they performed the procedure correctly (e.g. in a lab), but it is there to check things like: - Validity of the experimental setup - Validity of the statistical analysis methodology - Validity of the conclusions Take a look at the items in this comment:[1] > Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses All these can be caught with peer review. > together with an obsession for pursuing fashionable trends of dubious importance In contrast, this problem is intensified with peer review. > In their quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world. Or they retrofit hypotheses to fit their data. Not the role of peer review to catch these, but making your data and analyses scripts open will allow these to be caught by anyone who wishes to. The problem we have with current publishing is that I could use this technique to get faulty results, publish in a prestigious journal, get a 100 citations on my paper, and one of those citations will be from the person who looked at my data and saw obvious biases in my selection (which cannot be caught by a mere peer review). That one citation pointing out how wrong I am is lost in the noise. We need a way to highlight that citation - merely making a simple graph of connections via references doesn't get us there. [1] https://news.ycombinator.com/item?id=36140540 https://news.ycombinator.com/item?id=36140540
Fomite 3y ago
While you should never treat anything in the literature as having "a guarantee of correct results", it should be a careful first check.

gowld 3y ago

Most people have the wrond idea about what publication is, too. For almost all papers, for almost all readers, it doesn't matter if results are correct. It's just part of the game of academia. If you need to build something based on a paper, and you need it to work correctly (so, outside of public policy or macroeconomics), you need to do the work yourself. But if you just want to cite the work and write your own paper on top of it, go ahead. Published science is like ChatGPT ;-)

slt2021 3y ago
scientific publication is like a JIRA ticket. to get visibility of your work you need to close JIRA tickets, the more the better, since people look at aggregate metrics. The more your JIRA tickets are quoted in company documentation the better. but JIRA ticket is not code, in the same way publication is not science

caddemon 3y ago

It's never going to be a guarantee, but it's also implemented very shittily at present. There's a difference between what peer review is and what it should be. It makes sense to look at it realistically in day to day practice, but in discussion of systemic problems it should be viewed in a different light.

BeetleB 3y ago

Peer review cannot catch if they performed the procedure correctly (e.g. in a lab), but it is there to check things like: - Validity of the experimental setup - Validity of the statistical analysis methodology - Validity of the conclusions Take a look at the items in this comment:[1] > Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses All these can be caught with peer review. > together with an obsession for pursuing fashionable trends of dubious importance In contrast, this problem is intensified with peer review. > In their quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world. Or they retrofit hypotheses to fit their data. Not the role of peer review to catch these, but making your data and analyses scripts open will allow these to be caught by anyone who wishes to. The problem we have with current publishing is that I could use this technique to get faulty results, publish in a prestigious journal, get a 100 citations on my paper, and one of those citations will be from the person who looked at my data and saw obvious biases in my selection (which cannot be caught by a mere peer review). That one citation pointing out how wrong I am is lost in the noise. We need a way to highlight that citation - merely making a simple graph of connections via references doesn't get us there. [1] https://news.ycombinator.com/item?id=36140540 https://news.ycombinator.com/item?id=36140540

Fomite 3y ago

While you should never treat anything in the literature as having "a guarantee of correct results", it should be a careful first check.