4 ms·

Whenever this kind of thing happens, you get a lot of people advocating for the abolition of peer review and its replacement with Twitter or something. I simply

by knzhou 6y ago

Whenever this kind of thing happens, you get a lot of people advocating for the abolition of peer review and its replacement with Twitter or something. I simply don't understand that. Normal peer review means "we showed our work to about 2 people who know what they're talking about and they thought it was okay." Preprints mean "nobody who knows what they're talking about has looked at this carefully yet, but we think it's okay." Viral Twitter threads mean "here's my opinion, which hasn't been shown to anybody who actually knows what they're talking about, but which has been selected for maximum public response." Of course, in all cases, real credibility only comes once a paper has been looked over by hundreds of people who know what they're talking about. Yes, showing something to only 2 experts (who themselves are very busy, and are powerless to directly verify many of the paper's claims) is not a perfect initial quality cut -- but no cut at all is worse. If you really believe in the power of Twitter, take a long, hard look at the sensational, outrageous tweets you've liked. Go look at a sample of 100 old threads and see how many actually held up. The accuracy rate will not even be comparable to the Lancet's.

stephc_int13 6y ago
The credibility problem has been partially solved with a PageRank type algorithm. (Sergei and Larry were actually inspired by the weighting system of research papers) Highly regarded scientific journals like the Lancet are used to accelerate this weighting process. The problem is that nowadays they seem kind of outdated, opaque and corrupt on many levels. Nobody wants to suppress peer-review, simply replace the journals mafia with something less corrupt.
- knzhou 6y ago
  But this isn't a case of a journal acting like a "mafia". This is a case where something bad slipped past peer review, because of the inherent limitations of peer review. The reason is that expert time and attention is intrinsically scarce (if you double the number of experts, you double the number of papers produced, and hence double the amount of time that must be spent on peer review), and really properly vetting a paper takes a massive amount of time. This wouldn't be changed by system reform. In my opinion, the best option for everybody who's serious about this is to read a lot of papers, until you become "well-calibrated", i.e. when you have a good handle on the overall chances a preprint is true, vs. a Twitter thread, vs. a paper in a top-tier journal. This has to be learned from experience. Reform doesn't change the fact that you need to do this -- it just adds yet another venue, either more or less reliable than the previous ones, that you also need to become calibrated on.
  - fragmede 6y ago
    > read a lot of papers In order to do this, those that are serious need to be able to access papers, but paying hundreds or thousands of dollars for Elsevier can put them out of reach.
    - knzhou 6y ago
      Indeed, but that is the problem that preprints can solve. (As in, they don't solve the problem of telling what is correct, because that's just inherently hard, but they solve the problem of giving access so people can learn to tell.)
  - stephc_int13 6y ago
    There is also the problem with cost and access. (Remember Aaron Swartz) https://en.wikipedia.org/wiki/Open_access https://en.wikipedia.org/wiki/Open_access https://en.wikipedia.org/wiki/Aaron_Swartz https://en.wikipedia.org/wiki/Aaron_Swartz
skeletor_999 6y ago
I wouldn't do away with it, but I would encourage other scientists to engage with non-solicited peer review. A few sites have tried to encourage the wider scientific community to comment and debate published papers, but the overall participation remains low. It's unfortunate because I would be more confident in 100 peer reviewers post-publication than 2-3 that reviewed it for the journal. This already goes on in blogs and social media, but the same scientists haven't embraced the academic venues for this.

by knzhou 6y ago

stephc_int13 6y ago

The credibility problem has been partially solved with a PageRank type algorithm. (Sergei and Larry were actually inspired by the weighting system of research papers) Highly regarded scientific journals like the Lancet are used to accelerate this weighting process. The problem is that nowadays they seem kind of outdated, opaque and corrupt on many levels. Nobody wants to suppress peer-review, simply replace the journals mafia with something less corrupt.

knzhou 6y ago
But this isn't a case of a journal acting like a "mafia". This is a case where something bad slipped past peer review, because of the inherent limitations of peer review. The reason is that expert time and attention is intrinsically scarce (if you double the number of experts, you double the number of papers produced, and hence double the amount of time that must be spent on peer review), and really properly vetting a paper takes a massive amount of time. This wouldn't be changed by system reform. In my opinion, the best option for everybody who's serious about this is to read a lot of papers, until you become "well-calibrated", i.e. when you have a good handle on the overall chances a preprint is true, vs. a Twitter thread, vs. a paper in a top-tier journal. This has to be learned from experience. Reform doesn't change the fact that you need to do this -- it just adds yet another venue, either more or less reliable than the previous ones, that you also need to become calibrated on.
- fragmede 6y ago
  > read a lot of papers In order to do this, those that are serious need to be able to access papers, but paying hundreds or thousands of dollars for Elsevier can put them out of reach.
  - knzhou 6y ago
    Indeed, but that is the problem that preprints can solve. (As in, they don't solve the problem of telling what is correct, because that's just inherently hard, but they solve the problem of giving access so people can learn to tell.)
- stephc_int13 6y ago
  There is also the problem with cost and access. (Remember Aaron Swartz) https://en.wikipedia.org/wiki/Open_access https://en.wikipedia.org/wiki/Open_access https://en.wikipedia.org/wiki/Aaron_Swartz https://en.wikipedia.org/wiki/Aaron_Swartz

skeletor_999 6y ago

I wouldn't do away with it, but I would encourage other scientists to engage with non-solicited peer review. A few sites have tried to encourage the wider scientific community to comment and debate published papers, but the overall participation remains low. It's unfortunate because I would be more confident in 100 peer reviewers post-publication than 2-3 that reviewed it for the journal. This already goes on in blogs and social media, but the same scientists haven't embraced the academic venues for this.