7 ms·

Hmm... well Peerage of Science was an overlay on top of the journals. It was basically a system the journals would pay for to mediate their own review. Maybe

Hmm... well Peerage of Science was an overlay on top of the journals. It was basically a system the journals would pay for to mediate their own review. Maybe similar tech-wise, but very different goals. > Speaking for myself, I don't trawl through preprint servers looking for papers to review. > As it is, editors have a hard time getting people to review, and even after agreeing to review, to get the review back in reasonable time. Could we change that culture? The hope would be that we could build a culture of people helping each other review, for the simple reason that they want that help themselves when it comes time to for them to publish. With the reputation being more of a bonus. Keep in mind, review here is purely editorial feedback. It is not refereeing - that happens post publish through the voting system. > How does it handle peer review rings, where authors collude with reviewers to get bogus positive reviews? This is already an issue now. See above - review is not refereeing. It could be an issue in refereeing in the votes for sure though. The hope is that the overall weight of the community would override those rings. And in the long term we might even be able to identify them through analysis of the database. If we see the same group of people always up voting each other's down voted papers, we can flag that and take corrective action. > It offers reviewers an incentive to write a review that the paper author likes. I expect if reviewer writes "this paper duplicates work by Penzias & Wilson (1965) and offers nothing new - reject" then the author will reject it. While "Fix the typo on page 4, otherwise, go ahead and publish" will be accepted - even if the paper really is a waste of time. > It's a lot easier to get surface-level proofreading responses than substantive domain-specific responses. That may turn out to be true, but the thought is that the post-publish refereeing system will help incentivize people to take critical feedback seriously at the review step. After all, your goal is to publish something that will be well received. If you get good constructive critical feedback and ignore it, you risk being downvoted to oblivion and losing reputation. > Why aren't up voters strongly encouraged to post a response explaining their upvote? Why should upvotes not be equally visible? > What's the negative consequence should someone not provide a response? It's been asked before and it's definitely on the table. It's something I'd love more feedback on - should responses be required for all votes? Just down votes? Or (as I'm defaulting to) should we follow the StackExchange model of just encouraging responses on down votes? Seems maybe you would argue in favor of requiring responses on all votes? What's the reasoning? > Any reason to not develop this as an overlay journal on top of existing pre-print servers? Thus reducing a lot of the costs? We don't want it to be an overlay, because we want to ultimately replace the system. But it could make a lot of sense to work with the existing pre-print servers. I see no reason why we couldn't start as a pre-print and gradually evolve towards being the final publish, as long as we're clear about the end goal so that we don't just become "yet another pre-print". > Why would anyone use this system given PubPeer, and PeerJ, and eLife, and overlay journals, and Publons, and all of the other attempts at breaking the current system, why would someone choose this one? - PubPeer is just a comment overlay, can't read the papers (as I understand it). - PeerJ is still using the manual model, hence overhead and fees to publish ($1200) rather than diamond open access. This creates equity in access to publishing issues ( a lot of people cannot afford those fees ). - Elife, similarly, charged $3000 per article. These fees create massive equity in access to publishing issues. - Publons I wasn't previously familiar with, but it seems to be about just highlighting review work done for the journals, and not anything to do with open access. None of the attempts to break the current system (that I've found) seem to really be able to replace the journals by doing (almost) everything the journals do. At least, not with out just being a journal and facing the overhead problems. The one thing this project doesn't do that the journals do is curate reviews/refereeing, but the idea is that crowdsourcing could actually work better in the long run - though it's definitely going to bring its own set of problems that will need to be dealt with. But thanks! I hadn't heard of Peerage of Science, and Publons sounds vaguely familiar but I wasn't really familiar with it. I'm constantly learning more about what other efforts have come before and currently exist (there are a lot of them). And thinking through all the questions and counter points is always very helpful!

eesmith 4y ago
> Could we change that culture? I don't see how this system can do that. I stopped caring about Stack Overflow up- and down-votes long ago, I've grown tired of reading stories about the background politics and voting for Wikipedia, I can easily see how the proposed system can be gamed, and I don't see how it can defend against systemic gender, racial, or other social biases. > for the simple reason that they want that help themselves when it comes time to for them to publish Which leads to collusion. "You scratch my back, I'll scratch yours." My reviews are anonymous. I want to make it harder to retaliate against me, and make it harder to use what authority I've gained to influence things, and I trust the editors to mediate. > review is not refereeing Shrug. Okay. The question is, why do people want to get points? If it's good to get points, then there will be collusion to get those points. > the post-publish refereeing system will help incentivize people to take critical feedback seriously I have not found that system to be all that effective. > Seems maybe you would argue in favor of requiring responses on all votes? That is not my argument. I don't know one way or the other. I observed that "transparency" is described as being important for negative votes, but with no reason given why it's not also important for positive ones. > We don't want it to be an overlay, because we want to ultimately replace the system. The advocacy objects to a system with "high fees", "paywalls", "pay-to-play", etc. Preprint servers don't have fees, paywall, or pay-to-play. Thus, there is no reason to suspect you want to replace preprint servers, nor reason for why it's important to do so. > None of the attempts to break the current system How much do you know about eLife and Michael Eisen? How hard is it to get a waiver for the $3000 fee? "To ensure that eLife’s publication fee is not a barrier to publication we therefore offer a simple way for authors to apply for a fee waiver." https://reviewer.elifesciences.org/author-guide/fees#elife-publication-fee-waiver-policy https://reviewer.elifesciences.org/author-guide/fees#elife-p... It's also hard understand what you mean by "the current system", since I didn't realize pre-print servers were part of it. eLife requires preprint publication to bioRxiv or medRxiv. > there are a lot of them My question isn't "how does someone well-informed in all the different options choose yours?" but rather "given that most people aren't well-informed about the different options, why would they think to publish using your project?"
counters 4y ago
> Could we change that culture? In short? No. You probably can't. I hate being so blunt because I love the ideas you're proposing and have advocate for many similar things when I was still pursuing an academic career and now from outside of it (albeit in a scientific community where there is a very blurred line except for institutional affiliations). But you're not addressing the fundamental problem: there's such an over-saturation of PhDs and so few tenure-track positions in so many fields that there is one and only one objective for an early-career researcher: publish, publish, publish. That's how you build the portfolio to get called back for campus interviews for professorships; that's how you build your network and slide into funded grants before building your own, with connections to the program office or review committee that will help you get said grant funded. Although there are many worthwhile activities for an early career researcher to engage in (and, thankfully, a slowly growing recognition of the value of these activities, such as open source development and data curation), only a small subset of them feed into the ultimate goal: a tenured professorship. The critical shortage of reviewers these days, at least in my field, is partly due to social dynamics since the pandemic, but also very critically due to the perceived lack of value that serving as a review provides while you're in the heat of the rat race. It doesn't sound like your a practicing research scientist (meaning: you work in a job where producing research and publishing on this content is a primary metric against which your success is measured), but you should really talk to some across a diversity of fields. It takes a _lot_ of effort to provide a useful review. Effort that instead could be funneled into ones own research, for greater impact on your career. And if you're the author of a good paper, you don't need the other reviewers to "help" you with the process. It can be tedious and annoying sometimes, but hell - what difference dos it make if _one_ of your papers is getting held up, you've got 6 others ready to go by the end of the year, right? The problem is that it's not a "culture" issue. It's the fundamental practice of the modern scientific enterprise. The incentive structures are highly warped - a problem that many of us run into time and time again when we try to push through open science and reproducible science initiatives (and I'd definitely classify your effort in that group). When the fundamental practice and institutions change, the culture will change. If you'd be willing to bend and constrain your idea to more of a "Reddit of peer-reviewed literature" and focus on expert intersectional communities like ScienceBlogs, you might have something really novel here. There's a ton of value in the open-sourced curation mechanism. I've been involved heavily with /r/science in the past, and while the ecosystem of sub-reddits surrounding it definitely do a good job of "curating" science, it's not really the same thing as what you're proposing. It's a definite niche would could have a lot of value. But don't expect a lot of scientists to buy-in when you're not offering them anything in return. And again, to make this very, very clear: if your project doesn't help an academic get tenure, I don't see why they'd participate.

eesmith 4y ago

> Could we change that culture? I don't see how this system can do that. I stopped caring about Stack Overflow up- and down-votes long ago, I've grown tired of reading stories about the background politics and voting for Wikipedia, I can easily see how the proposed system can be gamed, and I don't see how it can defend against systemic gender, racial, or other social biases. > for the simple reason that they want that help themselves when it comes time to for them to publish Which leads to collusion. "You scratch my back, I'll scratch yours." My reviews are anonymous. I want to make it harder to retaliate against me, and make it harder to use what authority I've gained to influence things, and I trust the editors to mediate. > review is not refereeing Shrug. Okay. The question is, why do people want to get points? If it's good to get points, then there will be collusion to get those points. > the post-publish refereeing system will help incentivize people to take critical feedback seriously I have not found that system to be all that effective. > Seems maybe you would argue in favor of requiring responses on all votes? That is not my argument. I don't know one way or the other. I observed that "transparency" is described as being important for negative votes, but with no reason given why it's not also important for positive ones. > We don't want it to be an overlay, because we want to ultimately replace the system. The advocacy objects to a system with "high fees", "paywalls", "pay-to-play", etc. Preprint servers don't have fees, paywall, or pay-to-play. Thus, there is no reason to suspect you want to replace preprint servers, nor reason for why it's important to do so. > None of the attempts to break the current system How much do you know about eLife and Michael Eisen? How hard is it to get a waiver for the $3000 fee? "To ensure that eLife’s publication fee is not a barrier to publication we therefore offer a simple way for authors to apply for a fee waiver." https://reviewer.elifesciences.org/author-guide/fees#elife-publication-fee-waiver-policy https://reviewer.elifesciences.org/author-guide/fees#elife-p... It's also hard understand what you mean by "the current system", since I didn't realize pre-print servers were part of it. eLife requires preprint publication to bioRxiv or medRxiv. > there are a lot of them My question isn't "how does someone well-informed in all the different options choose yours?" but rather "given that most people aren't well-informed about the different options, why would they think to publish using your project?"

counters 4y ago

> Could we change that culture? In short? No. You probably can't. I hate being so blunt because I love the ideas you're proposing and have advocate for many similar things when I was still pursuing an academic career and now from outside of it (albeit in a scientific community where there is a very blurred line except for institutional affiliations). But you're not addressing the fundamental problem: there's such an over-saturation of PhDs and so few tenure-track positions in so many fields that there is one and only one objective for an early-career researcher: publish, publish, publish. That's how you build the portfolio to get called back for campus interviews for professorships; that's how you build your network and slide into funded grants before building your own, with connections to the program office or review committee that will help you get said grant funded. Although there are many worthwhile activities for an early career researcher to engage in (and, thankfully, a slowly growing recognition of the value of these activities, such as open source development and data curation), only a small subset of them feed into the ultimate goal: a tenured professorship. The critical shortage of reviewers these days, at least in my field, is partly due to social dynamics since the pandemic, but also very critically due to the perceived lack of value that serving as a review provides while you're in the heat of the rat race. It doesn't sound like your a practicing research scientist (meaning: you work in a job where producing research and publishing on this content is a primary metric against which your success is measured), but you should really talk to some across a diversity of fields. It takes a _lot_ of effort to provide a useful review. Effort that instead could be funneled into ones own research, for greater impact on your career. And if you're the author of a good paper, you don't need the other reviewers to "help" you with the process. It can be tedious and annoying sometimes, but hell - what difference dos it make if _one_ of your papers is getting held up, you've got 6 others ready to go by the end of the year, right? The problem is that it's not a "culture" issue. It's the fundamental practice of the modern scientific enterprise. The incentive structures are highly warped - a problem that many of us run into time and time again when we try to push through open science and reproducible science initiatives (and I'd definitely classify your effort in that group). When the fundamental practice and institutions change, the culture will change. If you'd be willing to bend and constrain your idea to more of a "Reddit of peer-reviewed literature" and focus on expert intersectional communities like ScienceBlogs, you might have something really novel here. There's a ton of value in the open-sourced curation mechanism. I've been involved heavily with /r/science in the past, and while the ecosystem of sub-reddits surrounding it definitely do a good job of "curating" science, it's not really the same thing as what you're proposing. It's a definite niche would could have a lot of value. But don't expect a lot of scientists to buy-in when you're not offering them anything in return. And again, to make this very, very clear: if your project doesn't help an academic get tenure, I don't see why they'd participate.