3 ms·

To be fair, the responsibility for the replication crises is much trickier. It’s based on human motivations and poor science: 1. Outright fraud 2. Lack of ind

To be fair, the responsibility for the replication crises is much trickier. It’s based on human motivations and poor science: 1. Outright fraud 2. Lack of independent verification of results before building up a body of work resulting in a bunch of garbage. Contributes to making #1 “easy”. 3. Financial pressures to publish or perish contributing to 1 & 2. If peer review didn’t exist you’d still something similar about producing “recognized” results. This is also probably why we haven’t done any major breakthroughs in particle physics which now has a much longer term thinking phase to come up with anything. The biggest problem with peer review is actually the establishment of an orthodoxy which discourages itself being upended by controlling the career success of anyone who dare challenge it - you have to support the orthodoxy to get career success and you fail to get recognition if your idea challenges the peer reviewers ideas and careers. That being said, such pressures existed before, but at least you could try to build your own following and it was competing orthodoxies instead of a single one winning out.

lisper 2y ago
> Financial pressures to publish or perish But those only exist because of the current peer-review model. There is a huge non-linearity built in to the publication process that provides disproportionate rewards for conning a small number of people, fewer than half a dozen. That, combined with a presumption of trustworthiness, produces a perverse incentive for attempting such cons because they are very easy to pull off, much easier than actually doing good science.
- vlovich123 2y ago
  The rewards come from the grant model which I forgot to mention and also has similar problems to peer review. The stipulation to publish is a secondary effect. If peer review didn’t exist there would still be pressure to show that you somehow convinced the leading experts in the field.
  - lisper 2y ago
    > there would still be pressure to show that you somehow convinced the leading experts in the field And how do you tell who are the leading experts in the field?
    - vlovich123 2y ago
      It doesn’t matter. There’s plenty of ways. Tenure, those who get cited the most as contributing to other works, whoever manages to shmooze their way onto the grant review board, whoever has been doing “good” work in a space etc etc. If you think that peer review is the only way status is established, I’m afraid you’re failing to grok how humans work - we’re very status oriented and will come up with any mechanism to formalize and broadcast that status and academia and the scientific world is not immune from this.
      - lisper 2y ago
        > Tenure And how do you think that gets decided? > those who get cited the most And how do you get cited without first getting published in a peer-reviewed publication? > whoever manages to shmooze their way onto the grant review board Do you think it's possible to do that without a publication record? > whoever has been doing “good” work in a space As decided by whom? The point is that the current system is based on a small cadre of people assessing each other's work on the assumption that they are all competent and trustworthy. The bigger the community, the easier it becomes to game the system, and the bigger the incentives to do so, and so the less reliable traditional peer review becomes as a predictor of scientific quality. To say nothing of the sheer horrible inefficiency. It takes months to do something that should take days. If anything was ever ripe for disruption, it's peer review. BTW, here is an example of what happens when someone actually sets out to game the system and is fairly good at it: https://www.youtube.com/watch?v=ODgYbmmgOss https://www.youtube.com/watch?v=ODgYbmmgOss
        vlovich123 2y ago
        > And how do you think that gets decided? Tenure precedes peer review afaik which I think pretty obviously negates this question - humans established tenure somehow so whatever that mechanism was. Peer review as a concept is quite old (17th century) and what these guys did on discord is peer review and collaboration. I’m assuming you’re just using shorthand to refer to journal peer review which is the more recent phenomenon. > And how do you get cited without first getting published in a peer-reviewed publication? Citations exist independently of peer review. Not sure why you think you can’t have one without the other. Journals are certainly not the only source cited. For example, math I believe doesn’t even generally use journals and yet citations are going strong there. > Do you think it's possible to do that without a publication record? Possible? Of course. Pick 10 random bureaucrats and have them pick admissions at random. Good? Well, now you seem to be arguing the pro publication position as a way of coming up with a better review board. But anyway, yes obviously there are better ways of establishing a grant review board by trying to populate it with some amount of “freethinkers”). Were agreed that the peer review system sucks for all sorts of reasons but we’re now very far afield from what I was trying to correct which is that the replication crises has many origins and isn’t just the fault of peer reviews. You’d have it even if journals and publish or perish weren’t a thing.

lisper 2y ago

> Financial pressures to publish or perish But those only exist because of the current peer-review model. There is a huge non-linearity built in to the publication process that provides disproportionate rewards for conning a small number of people, fewer than half a dozen. That, combined with a presumption of trustworthiness, produces a perverse incentive for attempting such cons because they are very easy to pull off, much easier than actually doing good science.

vlovich123 2y ago
The rewards come from the grant model which I forgot to mention and also has similar problems to peer review. The stipulation to publish is a secondary effect. If peer review didn’t exist there would still be pressure to show that you somehow convinced the leading experts in the field.
- lisper 2y ago
  > there would still be pressure to show that you somehow convinced the leading experts in the field And how do you tell who are the leading experts in the field?
  - vlovich123 2y ago
    It doesn’t matter. There’s plenty of ways. Tenure, those who get cited the most as contributing to other works, whoever manages to shmooze their way onto the grant review board, whoever has been doing “good” work in a space etc etc. If you think that peer review is the only way status is established, I’m afraid you’re failing to grok how humans work - we’re very status oriented and will come up with any mechanism to formalize and broadcast that status and academia and the scientific world is not immune from this.
    - lisper 2y ago
      > Tenure And how do you think that gets decided? > those who get cited the most And how do you get cited without first getting published in a peer-reviewed publication? > whoever manages to shmooze their way onto the grant review board Do you think it's possible to do that without a publication record? > whoever has been doing “good” work in a space As decided by whom? The point is that the current system is based on a small cadre of people assessing each other's work on the assumption that they are all competent and trustworthy. The bigger the community, the easier it becomes to game the system, and the bigger the incentives to do so, and so the less reliable traditional peer review becomes as a predictor of scientific quality. To say nothing of the sheer horrible inefficiency. It takes months to do something that should take days. If anything was ever ripe for disruption, it's peer review. BTW, here is an example of what happens when someone actually sets out to game the system and is fairly good at it: https://www.youtube.com/watch?v=ODgYbmmgOss https://www.youtube.com/watch?v=ODgYbmmgOss
      - vlovich123 2y ago
        > And how do you think that gets decided? Tenure precedes peer review afaik which I think pretty obviously negates this question - humans established tenure somehow so whatever that mechanism was. Peer review as a concept is quite old (17th century) and what these guys did on discord is peer review and collaboration. I’m assuming you’re just using shorthand to refer to journal peer review which is the more recent phenomenon. > And how do you get cited without first getting published in a peer-reviewed publication? Citations exist independently of peer review. Not sure why you think you can’t have one without the other. Journals are certainly not the only source cited. For example, math I believe doesn’t even generally use journals and yet citations are going strong there. > Do you think it's possible to do that without a publication record? Possible? Of course. Pick 10 random bureaucrats and have them pick admissions at random. Good? Well, now you seem to be arguing the pro publication position as a way of coming up with a better review board. But anyway, yes obviously there are better ways of establishing a grant review board by trying to populate it with some amount of “freethinkers”). Were agreed that the peer review system sucks for all sorts of reasons but we’re now very far afield from what I was trying to correct which is that the replication crises has many origins and isn’t just the fault of peer reviews. You’d have it even if journals and publish or perish weren’t a thing.