3 ms·

I've found the entire peer review concept in academia to be extremely odd. I'm not entirely sure what problem it solves. It seems like you have one of two types

I've found the entire peer review concept in academia to be extremely odd. I'm not entirely sure what problem it solves. It seems like you have one of two types of people reading these articles: * People who are already specialists/familiar enough with concepts. They'll either call BS upfront or run experimentation themselves to validate results. * People who aren't specialists and will need to corroborate evidence against other sources. My entire life as a software engineer has been built blogs, forums, and discussion from "random" people doing exactly the above.

cqqxo4zV46cp 3y ago
Most of the world isn’t ‘about’ software engineering. Software has some attributes that not very many other fields do. And software people have a habit of thinking that their experiencing writing software grants them some sort of transferable expertise, which it certainly does not.
kelipso 3y ago
> run experimentation themselves to validate results Let's just breeze through an that like it's nothing haha. Also the people reading these peer reviewed articles range from new grad students to researchers with decades of experience. There are many ways to see whether an article is high quality of not, which includes peer review, the journal it's published in, the research lab that wrote the paper. Reading a paper itself is a multi hour ordeal and you want to have a decent idea that it's not a crap paper before diving in. Believing what the paper says is something of a gamble because you really cannot just replicate an experiment that easily. And you need to read many many many papers before you can start doing your own research. So you want lots of assurances that a paper is good quality.
- abhgh 3y ago
  Agree. With more compute-intensive experiments in ML, I am not sure if any reviewer re-runs experiments. The reviewing is already voluntary - who's going to pay out of their pocket for the compute? It's not feasible.
  - adastra22 3y ago
    Re-running experiments is out of scope for any reviewer. You're supposed to trust their data, but review that their conclusion is actually supported by the data, as shown.
    - abhgh 3y ago
      Yeah, I agree. But I said what I did to make the larger point that empirical studies are hard to vouch for without some level of trust on both sides. For ex if someone selects favorable datasets, metrics, "random" seeds, p-hacking etc.

cqqxo4zV46cp 3y ago

Most of the world isn’t ‘about’ software engineering. Software has some attributes that not very many other fields do. And software people have a habit of thinking that their experiencing writing software grants them some sort of transferable expertise, which it certainly does not.

kelipso 3y ago

> run experimentation themselves to validate results Let's just breeze through an that like it's nothing haha. Also the people reading these peer reviewed articles range from new grad students to researchers with decades of experience. There are many ways to see whether an article is high quality of not, which includes peer review, the journal it's published in, the research lab that wrote the paper. Reading a paper itself is a multi hour ordeal and you want to have a decent idea that it's not a crap paper before diving in. Believing what the paper says is something of a gamble because you really cannot just replicate an experiment that easily. And you need to read many many many papers before you can start doing your own research. So you want lots of assurances that a paper is good quality.

abhgh 3y ago
Agree. With more compute-intensive experiments in ML, I am not sure if any reviewer re-runs experiments. The reviewing is already voluntary - who's going to pay out of their pocket for the compute? It's not feasible.
- adastra22 3y ago
  Re-running experiments is out of scope for any reviewer. You're supposed to trust their data, but review that their conclusion is actually supported by the data, as shown.
  - abhgh 3y ago
    Yeah, I agree. But I said what I did to make the larger point that empirical studies are hard to vouch for without some level of trust on both sides. For ex if someone selects favorable datasets, metrics, "random" seeds, p-hacking etc.