5 ms·

Or, research just gets published online free of charge for everyone to access, and important work will prove itself over time by being discovered, discussed and

Or, research just gets published online free of charge for everyone to access, and important work will prove itself over time by being discovered, discussed and by becoming influential. If anyone wants to change something about an article (the writing, the structure of the paper, or anything else a reviewer might want to edit) they can just do it and publish a new version. If people like the new version better, good, if they don't they can read the old version. Peer review as a filter for publishing is terrible in a time when making a few megabytes of text and images accessible is literally free. If anyone wants to run a content aggregator (aka a Journal) they just do it. If they want to change something about the article before it's approved for the aggregator they can contact the authors or ask someone to review it or whatever. Just make it accessible.

rhelz 3y ago
Whether or not your papers pass peer review—-and which journals it is published in—are important criteria for hiring, tenure, whether your grants are funded, etc. If you get rid of peer review, it’s not science. It’s just a vanity press.
- bumby 3y ago
  >If you get rid of peer review, it’s not science. It’s just a vanity press. If you believe replicability is central to science, the current paradigm doesn't necessarily converge on science either. And when people are graded on how many publications they garner, it borders on turning publication into a symbol of status rather than one of science.
  - rhelz 3y ago
    > If you believe replicability is central to science. I do believe that, but it doesn’t matter what anybody believes, replicatable experiments and results, which peers can review and agree on, are the soul of science. Without that it’s not science, it’s just creative writing.
    - bumby 3y ago
      Then don’t you think the first part of that process is explaining the methods and results of your experiment? That’s precisely what the current situation does. What’s lacking is the replication incentives.
      - rhelz 3y ago
        I propose to supply those incentives—-pay for good reviewer and replicators. If you want something done, you gotta pay for it. We can’t just rely on volunteers.
        bumby 3y ago
        I'm on board with that idea, as long as we can also provide guardrails against the perverse incentives of paying for them. E.g., we need to avoid frivolous reviews/replication as well as something evolving into a "pay for a good review" service.
        rhelz 3y ago
        Well, if you are paying somebody to do something, you gain a lot of leverage: 1. You can negotiate a due date. No more waiting for years before the journal's reviewers actually review your paper. 2. You can negotiate a set of deliverables. You can specify that they can't just say "this sux"; they have to show the lines where the big hairy proof are wrong, or if its an algorithm, they have to actually code it up and run it themselves before they say it doesn't work. 3. You can more reliably attract good reviewers. If you aren't begging for people to volunteer, but you are paying good money, you can be a lot pickier about who you hire. I mean, I've been a consultant: what are the guardrails that I won't rip-off my clients? I don't want to ruin my reputation, I want repeat business, and I want to be able to charge a high hourly rate because I deliver a premium product. Same guardrails would apply to peer reviewers and to reproducers.
        bumby 3y ago
        Sure, but I’m poking at the bad leverage you can also wield. 1) you can create undue schedule pressure that results in a cursory review that may not catch the more nuanced problems in your investigation. 2) you can be more belligerent about not sharing data. If they want to get, they won’t argue. 3) you can pay for reviewers who you know will give a positive review. Without guards against this, it’s almost a certainty that the glut of PhDs will result in some treating it like a scam de hustle where it’s more about the economics than the science. Some consultants are well known to play the game where they tell clients what they want to hear rather than what they need to hear. I don’t think consultancy is a good model for this.
- geysersam 3y ago
  And the current system is a holy grail that can never change? You can still based your hiring based on reviews made by peers
ska 3y ago
> Just make it accessible. We already have that system, it's called the internet. Nothing stops you or I from putting our ideas online for all to read, comment on, update, etc. The role of the publishers, flawed as it is, has little to do with the physical cost of producing or providing an article, and is filling (one can argue badly) a role in curation and archival that is clearly needed. Any proposal to change the system really has to address how those roles are met, hopefully cheaper than currently but definitely not more expensive because mostly people don't get paid (in $ or career or anything) now - or has to provide a funding source for it. I don't really see how your outlined scenario addresses that, at least not in a way that's functionally different than today. Can you expand?
- paulmd 3y ago
  > We already have that system, it's called the internet. Nothing stops you or I from putting our ideas online for all to read, comment on, update, etc. Are you asking how arxiv is different from blogspot?
  - ska 3y ago
    No, although the mechanism for hosting the content aren’t that important. Preprint servers are very useful but haven’t replaced journals for good reasons.
    - staunton 3y ago
      What are those reasons? The only thing I see that journals do which preprint servers couldn't easily take over is prestige.
      - ska 3y ago
        You have the causality wrong. Prestige comes to journals by doing a good (or at least, better than peers) job of being a journal, which is providing a necessary function to the academic research process. If you want to improve on that system, you have to improve on those functions, or reduce the reliance on them by providing something better. Put it another way, if you can design a system with a better ROC curve for classifying research, with a better TP rate for good papers, and have it cost less in real terms that current academic papers, then you are on to something. If all you've got is "papers should be free" or "it's too hard to access publishing from the outside" what you have are complaints, not solutions.

rhelz 3y ago

Whether or not your papers pass peer review—-and which journals it is published in—are important criteria for hiring, tenure, whether your grants are funded, etc. If you get rid of peer review, it’s not science. It’s just a vanity press.

bumby 3y ago
>If you get rid of peer review, it’s not science. It’s just a vanity press. If you believe replicability is central to science, the current paradigm doesn't necessarily converge on science either. And when people are graded on how many publications they garner, it borders on turning publication into a symbol of status rather than one of science.
- rhelz 3y ago
  > If you believe replicability is central to science. I do believe that, but it doesn’t matter what anybody believes, replicatable experiments and results, which peers can review and agree on, are the soul of science. Without that it’s not science, it’s just creative writing.
  - bumby 3y ago
    Then don’t you think the first part of that process is explaining the methods and results of your experiment? That’s precisely what the current situation does. What’s lacking is the replication incentives.
    - rhelz 3y ago
      I propose to supply those incentives—-pay for good reviewer and replicators. If you want something done, you gotta pay for it. We can’t just rely on volunteers.
      - bumby 3y ago
        I'm on board with that idea, as long as we can also provide guardrails against the perverse incentives of paying for them. E.g., we need to avoid frivolous reviews/replication as well as something evolving into a "pay for a good review" service.
        rhelz 3y ago
        Well, if you are paying somebody to do something, you gain a lot of leverage: 1. You can negotiate a due date. No more waiting for years before the journal's reviewers actually review your paper. 2. You can negotiate a set of deliverables. You can specify that they can't just say "this sux"; they have to show the lines where the big hairy proof are wrong, or if its an algorithm, they have to actually code it up and run it themselves before they say it doesn't work. 3. You can more reliably attract good reviewers. If you aren't begging for people to volunteer, but you are paying good money, you can be a lot pickier about who you hire. I mean, I've been a consultant: what are the guardrails that I won't rip-off my clients? I don't want to ruin my reputation, I want repeat business, and I want to be able to charge a high hourly rate because I deliver a premium product. Same guardrails would apply to peer reviewers and to reproducers.
        bumby 3y ago
        Sure, but I’m poking at the bad leverage you can also wield. 1) you can create undue schedule pressure that results in a cursory review that may not catch the more nuanced problems in your investigation. 2) you can be more belligerent about not sharing data. If they want to get, they won’t argue. 3) you can pay for reviewers who you know will give a positive review. Without guards against this, it’s almost a certainty that the glut of PhDs will result in some treating it like a scam de hustle where it’s more about the economics than the science. Some consultants are well known to play the game where they tell clients what they want to hear rather than what they need to hear. I don’t think consultancy is a good model for this.
geysersam 3y ago
And the current system is a holy grail that can never change? You can still based your hiring based on reviews made by peers

ska 3y ago

> Just make it accessible. We already have that system, it's called the internet. Nothing stops you or I from putting our ideas online for all to read, comment on, update, etc. The role of the publishers, flawed as it is, has little to do with the physical cost of producing or providing an article, and is filling (one can argue badly) a role in curation and archival that is clearly needed. Any proposal to change the system really has to address how those roles are met, hopefully cheaper than currently but definitely not more expensive because mostly people don't get paid (in $ or career or anything) now - or has to provide a funding source for it. I don't really see how your outlined scenario addresses that, at least not in a way that's functionally different than today. Can you expand?

paulmd 3y ago
> We already have that system, it's called the internet. Nothing stops you or I from putting our ideas online for all to read, comment on, update, etc. Are you asking how arxiv is different from blogspot?
- ska 3y ago
  No, although the mechanism for hosting the content aren’t that important. Preprint servers are very useful but haven’t replaced journals for good reasons.
  - staunton 3y ago
    What are those reasons? The only thing I see that journals do which preprint servers couldn't easily take over is prestige.
    - ska 3y ago
      You have the causality wrong. Prestige comes to journals by doing a good (or at least, better than peers) job of being a journal, which is providing a necessary function to the academic research process. If you want to improve on that system, you have to improve on those functions, or reduce the reliance on them by providing something better. Put it another way, if you can design a system with a better ROC curve for classifying research, with a better TP rate for good papers, and have it cost less in real terms that current academic papers, then you are on to something. If all you've got is "papers should be free" or "it's too hard to access publishing from the outside" what you have are complaints, not solutions.