4 ms·

That sounds (as I said before) quite specific to the field you found yourself in. That isn't how things work everywhere. For a relatively recent example - in m

That sounds (as I said before) quite specific to the field you found yourself in. That isn't how things work everywhere. For a relatively recent example - in my field of quantum information someone found in the course of reviewing some newer work that there was a mistake in one of the big foundational papers that this newer work was based on. They shared it with some of their colleagues and together they did some work finding out exactly where the error was and what the impact is on work that is based on it. The response from the community was they got a talk slot at a pretty prestigious conference (QIP 2023, you can watch the video here: https://www.youtube.com/watch?v=2Xyodvh6DSY https://www.youtube.com/watch?v=2Xyodvh6DSY) and the work they did has been published here https://quantum-journal.org/papers/q-2023-09-07-1103/ https://quantum-journal.org/papers/q-2023-09-07-1103/ and in a followup paper here https://www.nature.com/articles/s41567-023-02289-9 https://www.nature.com/articles/s41567-023-02289-9. My point is essentially that it isn't a big shock that this error wasn't found by peer review when the foundational paper was published in 2008. Even when it is working as well as possible peer review can only detect most errors, not all errors. However when your field is functioning "normally" and everyone does their jobs properly this doesn't matter because the errors that get past peer review get detected by other people building on the work later. On the other hand if your field is not functioning properly and people are knowingly doing stuff that is wrong then peer review can't help you. If the status of the field is so broken that people who know stuff is wrong are still pushing for it to be published then you aren't in a place where peer review can save you. My opinion is you are incorrectly putting the blame on the (highly imperfect) process of peer review here where it doesn't really belong. Peer review as we perform it today sucks for many reasons, but it isn't to blame for the problems you saw.

PaulHoule 3y ago
It's not to blame for these problems but it fails to counter them. In medicine see https://www.cochranelibrary.com/ https://www.cochranelibrary.com/ where there is a systematic process to search the literature and select papers with usable results. I've never seen more than 50% of papers get accepted and the other day I saw one where they found 80 invalid papers and 2 valid papers. It seems to me that peer review isn't accomplishing very much if it is passing through papers that can't be built on. For that matter, why is this work even being funded?
- eigenket 3y ago
  Yeah I wouldn't expect it to counter such problems. Peer review is by its nature review by other people in the same field. If you have a field where "bogus" methods are being used in thousands of papers then you aren't going to be able to get things reviewed properly. This isn't an indictment of peer review as it is practised in every field.
  - PaulHoule 3y ago
    I’ll say a good field is an island in an ocean of bad fields. Look at hep-th. If in 100 years people conclude there is no evidence for supersymmetry and extra dimensions something like 90% of that field will be “not even wrong”. There is so much of the drunk looking for his lost keys under a lamp. I mean, there is a huge industry in 2-d gravity where it is possible to calculate things but gravity doesn’t attract in 2-d so the line between that and Newton’s apple is unclear. I’ve talked to astro-ph refugees who think that current theories about accretion disks are “not even wrong”. Then there are all the folks who tell me firewalls make no sense because there is no such thing as the event horizon (locally) but from the viewpoint of people who fall in it is a tiny difference getting killed at the apparent horizon meanwhile the nonsensical “black hole information paradox” (sorry, no unitarity = no quantum mechanics) still gets breathless and insufferable articles written about it just because somebody famous for their disability made a bet. If young people were able to get established in the field you wouldn’t see people so fixated on bad ideas.

PaulHoule 3y ago

It's not to blame for these problems but it fails to counter them. In medicine see https://www.cochranelibrary.com/ https://www.cochranelibrary.com/ where there is a systematic process to search the literature and select papers with usable results. I've never seen more than 50% of papers get accepted and the other day I saw one where they found 80 invalid papers and 2 valid papers. It seems to me that peer review isn't accomplishing very much if it is passing through papers that can't be built on. For that matter, why is this work even being funded?

eigenket 3y ago
Yeah I wouldn't expect it to counter such problems. Peer review is by its nature review by other people in the same field. If you have a field where "bogus" methods are being used in thousands of papers then you aren't going to be able to get things reviewed properly. This isn't an indictment of peer review as it is practised in every field.
- PaulHoule 3y ago
  I’ll say a good field is an island in an ocean of bad fields. Look at hep-th. If in 100 years people conclude there is no evidence for supersymmetry and extra dimensions something like 90% of that field will be “not even wrong”. There is so much of the drunk looking for his lost keys under a lamp. I mean, there is a huge industry in 2-d gravity where it is possible to calculate things but gravity doesn’t attract in 2-d so the line between that and Newton’s apple is unclear. I’ve talked to astro-ph refugees who think that current theories about accretion disks are “not even wrong”. Then there are all the folks who tell me firewalls make no sense because there is no such thing as the event horizon (locally) but from the viewpoint of people who fall in it is a tiny difference getting killed at the apparent horizon meanwhile the nonsensical “black hole information paradox” (sorry, no unitarity = no quantum mechanics) still gets breathless and insufferable articles written about it just because somebody famous for their disability made a bet. If young people were able to get established in the field you wouldn’t see people so fixated on bad ideas.