3 ms·

> Who's going to replicate an experiment that takes a $5 billion particle accelerator? In the case of the LHC, which I assume is what you're talking about, thi

> Who's going to replicate an experiment that takes a $5 billion particle accelerator? In the case of the LHC, which I assume is what you're talking about, this is why we have 4 pretty much independent detectors around the LHC looking at different but at least partially overlapping stuff. The Higgs boson was detected by the ATLAS detector, but wasn't considered to be "discovered" until it had also been detected at the CMS detector. > Peer review is an absolute joke That strongly depends on the field and the journal in question.

PaulHoule 3y ago
If they flew out the peer reviewer to interview the researcher and take a look at their apparatus or if the peer reviewer loaded up a Jupyter notebook and rechecked the analysis there might be some value in peer review. If peer review is just somebody looking at the paper superficially than it’s of very limited worth. When I wrote my first paper somebody else had done a very similar experiment at another university and we wound method up peer reviewing each other’s papers. Turned out we were using the same bogus data analysis that was being used in 1000s of other papers. I was vaguely aware of this when I wrote the paper but did not feel psychologically safe at all in an environment where the American Physical Society claimed a PhD had a 3% chance of having a permanent career in the field. Someone who was later to become a titan in the field was at our lab and was crying some nights because he had no idea where he was going to get his next postdoc wrote a paper about this problem more than a decade later after he had gotten tenure in a statistics journal that physicists probably won’t even read.
- eigenket 3y ago
  That sounds (as I said before) quite specific to the field you found yourself in. That isn't how things work everywhere. For a relatively recent example - in my field of quantum information someone found in the course of reviewing some newer work that there was a mistake in one of the big foundational papers that this newer work was based on. They shared it with some of their colleagues and together they did some work finding out exactly where the error was and what the impact is on work that is based on it. The response from the community was they got a talk slot at a pretty prestigious conference (QIP 2023, you can watch the video here: https://www.youtube.com/watch?v=2Xyodvh6DSY https://www.youtube.com/watch?v=2Xyodvh6DSY) and the work they did has been published here https://quantum-journal.org/papers/q-2023-09-07-1103/ https://quantum-journal.org/papers/q-2023-09-07-1103/ and in a followup paper here https://www.nature.com/articles/s41567-023-02289-9 https://www.nature.com/articles/s41567-023-02289-9. My point is essentially that it isn't a big shock that this error wasn't found by peer review when the foundational paper was published in 2008. Even when it is working as well as possible peer review can only detect most errors, not all errors. However when your field is functioning "normally" and everyone does their jobs properly this doesn't matter because the errors that get past peer review get detected by other people building on the work later. On the other hand if your field is not functioning properly and people are knowingly doing stuff that is wrong then peer review can't help you. If the status of the field is so broken that people who know stuff is wrong are still pushing for it to be published then you aren't in a place where peer review can save you. My opinion is you are incorrectly putting the blame on the (highly imperfect) process of peer review here where it doesn't really belong. Peer review as we perform it today sucks for many reasons, but it isn't to blame for the problems you saw.
  - PaulHoule 3y ago
    It's not to blame for these problems but it fails to counter them. In medicine see https://www.cochranelibrary.com/ https://www.cochranelibrary.com/ where there is a systematic process to search the literature and select papers with usable results. I've never seen more than 50% of papers get accepted and the other day I saw one where they found 80 invalid papers and 2 valid papers. It seems to me that peer review isn't accomplishing very much if it is passing through papers that can't be built on. For that matter, why is this work even being funded?
    - eigenket 3y ago
      Yeah I wouldn't expect it to counter such problems. Peer review is by its nature review by other people in the same field. If you have a field where "bogus" methods are being used in thousands of papers then you aren't going to be able to get things reviewed properly. This isn't an indictment of peer review as it is practised in every field.
      - PaulHoule 3y ago
        I’ll say a good field is an island in an ocean of bad fields. Look at hep-th. If in 100 years people conclude there is no evidence for supersymmetry and extra dimensions something like 90% of that field will be “not even wrong”. There is so much of the drunk looking for his lost keys under a lamp. I mean, there is a huge industry in 2-d gravity where it is possible to calculate things but gravity doesn’t attract in 2-d so the line between that and Newton’s apple is unclear. I’ve talked to astro-ph refugees who think that current theories about accretion disks are “not even wrong”. Then there are all the folks who tell me firewalls make no sense because there is no such thing as the event horizon (locally) but from the viewpoint of people who fall in it is a tiny difference getting killed at the apparent horizon meanwhile the nonsensical “black hole information paradox” (sorry, no unitarity = no quantum mechanics) still gets breathless and insufferable articles written about it just because somebody famous for their disability made a bet. If young people were able to get established in the field you wouldn’t see people so fixated on bad ideas.
pa7x1 3y ago
And the same was done for LIGO and the discovery of gravitational waves. 2 observatories, one in Hanford and one in Livingstone.
mcmoor 3y ago
Yeah if one tried to find rot in academics, I think applied particle physics is one of the furthest from it. The scientists are super allergic to unfalsifiable theories that they start to doubt themselves when prediction is off by nanometers.

PaulHoule 3y ago

If they flew out the peer reviewer to interview the researcher and take a look at their apparatus or if the peer reviewer loaded up a Jupyter notebook and rechecked the analysis there might be some value in peer review. If peer review is just somebody looking at the paper superficially than it’s of very limited worth. When I wrote my first paper somebody else had done a very similar experiment at another university and we wound method up peer reviewing each other’s papers. Turned out we were using the same bogus data analysis that was being used in 1000s of other papers. I was vaguely aware of this when I wrote the paper but did not feel psychologically safe at all in an environment where the American Physical Society claimed a PhD had a 3% chance of having a permanent career in the field. Someone who was later to become a titan in the field was at our lab and was crying some nights because he had no idea where he was going to get his next postdoc wrote a paper about this problem more than a decade later after he had gotten tenure in a statistics journal that physicists probably won’t even read.

eigenket 3y ago
That sounds (as I said before) quite specific to the field you found yourself in. That isn't how things work everywhere. For a relatively recent example - in my field of quantum information someone found in the course of reviewing some newer work that there was a mistake in one of the big foundational papers that this newer work was based on. They shared it with some of their colleagues and together they did some work finding out exactly where the error was and what the impact is on work that is based on it. The response from the community was they got a talk slot at a pretty prestigious conference (QIP 2023, you can watch the video here: https://www.youtube.com/watch?v=2Xyodvh6DSY https://www.youtube.com/watch?v=2Xyodvh6DSY) and the work they did has been published here https://quantum-journal.org/papers/q-2023-09-07-1103/ https://quantum-journal.org/papers/q-2023-09-07-1103/ and in a followup paper here https://www.nature.com/articles/s41567-023-02289-9 https://www.nature.com/articles/s41567-023-02289-9. My point is essentially that it isn't a big shock that this error wasn't found by peer review when the foundational paper was published in 2008. Even when it is working as well as possible peer review can only detect most errors, not all errors. However when your field is functioning "normally" and everyone does their jobs properly this doesn't matter because the errors that get past peer review get detected by other people building on the work later. On the other hand if your field is not functioning properly and people are knowingly doing stuff that is wrong then peer review can't help you. If the status of the field is so broken that people who know stuff is wrong are still pushing for it to be published then you aren't in a place where peer review can save you. My opinion is you are incorrectly putting the blame on the (highly imperfect) process of peer review here where it doesn't really belong. Peer review as we perform it today sucks for many reasons, but it isn't to blame for the problems you saw.
- PaulHoule 3y ago
  It's not to blame for these problems but it fails to counter them. In medicine see https://www.cochranelibrary.com/ https://www.cochranelibrary.com/ where there is a systematic process to search the literature and select papers with usable results. I've never seen more than 50% of papers get accepted and the other day I saw one where they found 80 invalid papers and 2 valid papers. It seems to me that peer review isn't accomplishing very much if it is passing through papers that can't be built on. For that matter, why is this work even being funded?
  - eigenket 3y ago
    Yeah I wouldn't expect it to counter such problems. Peer review is by its nature review by other people in the same field. If you have a field where "bogus" methods are being used in thousands of papers then you aren't going to be able to get things reviewed properly. This isn't an indictment of peer review as it is practised in every field.
    - PaulHoule 3y ago
      I’ll say a good field is an island in an ocean of bad fields. Look at hep-th. If in 100 years people conclude there is no evidence for supersymmetry and extra dimensions something like 90% of that field will be “not even wrong”. There is so much of the drunk looking for his lost keys under a lamp. I mean, there is a huge industry in 2-d gravity where it is possible to calculate things but gravity doesn’t attract in 2-d so the line between that and Newton’s apple is unclear. I’ve talked to astro-ph refugees who think that current theories about accretion disks are “not even wrong”. Then there are all the folks who tell me firewalls make no sense because there is no such thing as the event horizon (locally) but from the viewpoint of people who fall in it is a tiny difference getting killed at the apparent horizon meanwhile the nonsensical “black hole information paradox” (sorry, no unitarity = no quantum mechanics) still gets breathless and insufferable articles written about it just because somebody famous for their disability made a bet. If young people were able to get established in the field you wouldn’t see people so fixated on bad ideas.

pa7x1 3y ago

And the same was done for LIGO and the discovery of gravitational waves. 2 observatories, one in Hanford and one in Livingstone.

mcmoor 3y ago

Yeah if one tried to find rot in academics, I think applied particle physics is one of the furthest from it. The scientists are super allergic to unfalsifiable theories that they start to doubt themselves when prediction is off by nanometers.