4 ms·

I don't want to quibble with your experiences of peer review; your experiences are valid. But what does a journal provide, then, exactly? Are they relics of the

I don't want to quibble with your experiences of peer review; your experiences are valid. But what does a journal provide, then, exactly? Are they relics of the era before cheap self-publication?

tptacek 6y ago
"A panel of experts from this field read this paper and felt it likely to make significant contributions". For high-impact journals, you can add "so much so that it was selected to be among the 25% of submissions this cycle to be published". What peer review does not say is "a panel of experts carefully vetted this paper to ensure that its conclusions are accurate". In fact, part of the premise of replication projects is that peer review doesn't say that. I've only been a reviewer a couple times, and only in computer science. Other people on HN have experience reviewing for other hard science venues. Maybe their experiences are different. But I think it's notable that you don't hear that in comments about what this hoax exposed; has there been any HN comment from a reviewer saying that they were expected to rigorously vet submissions, the way a PhD board does with a thesis? Daniel Engber makes many of these points better than I do, here: https://slate.com/technology/2018/10/grievance-studies-hoax-not-academic-scandal.html https://slate.com/technology/2018/10/grievance-studies-hoax-...
- thu2111 6y ago
  The problem is not so much the system as how it's presented: the media presents "peer-reviewed study" as a gold standard, practically synonymous with correct. And unfortunately academia absolutely encourages this practice. The message that peer review is largely meaningless hasn't really been made by academia, for obvious reasons, and thus stupid papers getting accepted by journals will continue to be interesting and news-worthy until people get the message that peer review doesn't mean much. The world will then move on to "replicated" as a gold standard. This will be better but not by much. Just a few weeks ago Imperial College London published a press release claiming their Report 9 results from their COVID-19 simulator program had been replicated. Worth a press release because after it was open sourced it was discovered to be filled with non-deterministic behaviour, even with fixed RNG seeds. Unfortunately the press release was fraudulent. The report they cited as evidence of replication was by a friendly academic. He said he was able to replicate the results, then admitted every number he got out was different, some by 10%-25% different. This is the output of a computer simulation so the allowable difference is 0%. Despite this, Nature and other outlets proceeded to report that ICL's COVID model was "replicated". In the end, academia is strongly incentivised to appear credible, not actually be so. Being credible requires you can reliably turn your findings into something useful, as when in the realm of pure theory you can never be sure your findings actually hold up in reality. Corporate research has this attribute (eventually), academic research doesn't, as academics are rewarded for writing clever sounding papers not being correct.
  - tptacek 6y ago
    This seems totally fair!
chalst 6y ago
I'm inclined to say the biggest piece of value peer review adds is that it ensures that new research takes into account the most relevant past research. It also means that the argumentation in most peer-reviewed research passes appropriate "smell tests", which is rather weaker than saying the argumentation is sound.

I don't want to quibble with your experiences of peer review; your experiences are valid. But what does a journal provide, then, exactly? Are they relics of the era before cheap self-publication?

tptacek 6y ago

"A panel of experts from this field read this paper and felt it likely to make significant contributions". For high-impact journals, you can add "so much so that it was selected to be among the 25% of submissions this cycle to be published". What peer review does not say is "a panel of experts carefully vetted this paper to ensure that its conclusions are accurate". In fact, part of the premise of replication projects is that peer review doesn't say that. I've only been a reviewer a couple times, and only in computer science. Other people on HN have experience reviewing for other hard science venues. Maybe their experiences are different. But I think it's notable that you don't hear that in comments about what this hoax exposed; has there been any HN comment from a reviewer saying that they were expected to rigorously vet submissions, the way a PhD board does with a thesis? Daniel Engber makes many of these points better than I do, here: https://slate.com/technology/2018/10/grievance-studies-hoax-not-academic-scandal.html https://slate.com/technology/2018/10/grievance-studies-hoax-...

thu2111 6y ago
The problem is not so much the system as how it's presented: the media presents "peer-reviewed study" as a gold standard, practically synonymous with correct. And unfortunately academia absolutely encourages this practice. The message that peer review is largely meaningless hasn't really been made by academia, for obvious reasons, and thus stupid papers getting accepted by journals will continue to be interesting and news-worthy until people get the message that peer review doesn't mean much. The world will then move on to "replicated" as a gold standard. This will be better but not by much. Just a few weeks ago Imperial College London published a press release claiming their Report 9 results from their COVID-19 simulator program had been replicated. Worth a press release because after it was open sourced it was discovered to be filled with non-deterministic behaviour, even with fixed RNG seeds. Unfortunately the press release was fraudulent. The report they cited as evidence of replication was by a friendly academic. He said he was able to replicate the results, then admitted every number he got out was different, some by 10%-25% different. This is the output of a computer simulation so the allowable difference is 0%. Despite this, Nature and other outlets proceeded to report that ICL's COVID model was "replicated". In the end, academia is strongly incentivised to appear credible, not actually be so. Being credible requires you can reliably turn your findings into something useful, as when in the realm of pure theory you can never be sure your findings actually hold up in reality. Corporate research has this attribute (eventually), academic research doesn't, as academics are rewarded for writing clever sounding papers not being correct.
- tptacek 6y ago
  This seems totally fair!

chalst 6y ago

I'm inclined to say the biggest piece of value peer review adds is that it ensures that new research takes into account the most relevant past research. It also means that the argumentation in most peer-reviewed research passes appropriate "smell tests", which is rather weaker than saying the argumentation is sound.