4 ms·

Hey HN, I've had this idea for an open scientific web platform for years. It seemed like such an obvious thing to try. I kept waiting for someone else to buil

Hey HN, I've had this idea for an open scientific web platform for years. It seemed like such an obvious thing to try. I kept waiting for someone else to build it, but to my knowledge no one else has. I finally saved up enough to take some time away from work and build a proof of concept. I'm looking for feedback on the PoC from scientists, scholars, researchers and academics to try to figure out if there is merit to this idea. I would deeply appreciate any and all feedback, and help getting this request in front of more of the target audience.

ISL 4y ago
Consider finding an academic niche in which to really make this product sing; a community that is receptive to the idea. If the PoC can be shown to work in one community, neighboring fields can then point at it and say, "see, this thing is working over in community X. Journal Y is going to take three months to get this reviewed, and this paper is kind of a throwaway -- let's try X and see how it goes." arXiv started in a narrow niche that absolutely needed it, perhaps the same will happen here. Long-term, I'm not sure how this system will pay for itself. Keep costs low and dig in for the long haul. Academia can have very low switching costs (a professor can try your service on a whim), but has enough inertia that it can take decades to turn the ship (deans need to know a prospective hire's impact factor in top-tier journals). Also, as a referee -- pay referees what their time is worth. For real. You'll get real good referees if you do.
- Test0129 4y ago
  The other side of it is reputation. Academic publishing to a questionable journal (even if it's a good journal) can be a black mark on your career. You not only have to get over the hump of at least break even profit but also convince tenured professors stuck in their ways that your journal will gain and keep a positive reputation. This task is insurmountable for the most part. At least with the current way academia works. Current journals have been around for decades, sometimes even reaching the 100 year mark. Back in the ye olde days it was easy for anyone to publish anything (even non-academics). These days these journals are so deeply entrenched in the minds of academics it would be like convincing 10% of fortune 500 companies to switch from excel to your new cool software. Even it's it better they won't because of this mental block.
  - dbingham 4y ago
    Yeah, that may be. But it seems worth trying given all the negative impacts of the status quo.
    - Test0129 4y ago
      I'm not saying to _not_ try it. It's just going to be a dumpster-fire-money-pit that has an extremely high probability of either bankrupting you or producing nothing of significant value. This isn't a shot at OP. The best way to force change here isn't yet another journal, but rather fix the access issues of current journals, convince them it's a good idea to be less exclusive (ever noticed certain initials tend to be published more than others?), and moreover stop the university-journal death-bond that currently is in the market. Good luck at any rate.
  - blep_ 4y ago
    Could the solution be "don't actually tie it to the journal"? Have the journal treat itself more as an implementation detail than as an end in itself; you didn't publish a paper in the Journal of Lasers, you just published a paper about lasers. This is still a mental shift for academics to get over, but I wonder if it's a more feasible one.
- dbingham 4y ago
  > Consider finding an academic niche in which to really make this product sing; a community that is receptive to the idea. If the PoC can be shown to work in one community, neighboring fields can then point at it and say, "see, this thing is working over in community X. Journal Y is going to take three months to get this reviewed, and this paper is kind of a throwaway -- let's try X and see how it goes." > arXiv started in a narrow niche that absolutely needed it, perhaps the same will happen here. That's very similar to my thinking - I'm actually hoping to start with the file drawer problem. When we're ready for open beta, put out a call for file drawered papers. It's also been suggested that we start as pre-print, while being clear about the ultimate intentions to become a journal, and go from there. Maybe even reach out to the existing pre-print servers to partner with them in some way. > Long-term, I'm not sure how this system will pay for itself. Keep costs low and dig in for the long haul. Academia can have very low switching costs (a professor can try your service on a whim), but has enough inertia that it can take decades to turn the ship (deans need to know a prospective hire's impact factor in top-tier journals). Noted. The thought is that it could be funded through donations, grants, and eventually by the institutions if it gains enough traction. But yeah, it's becoming clear that grants at the beginning may be the only way. We'll see how much we get in terms of donations once we hit open beta. > Also, as a referee -- pay referees what their time is worth. For real. You'll get real good referees if you do. ...well... the idea is actually to crowd source refereeing and review. And for people to do it because, well, it needs done. Everyone wants good reviews and everyone knows refereeing needs done, so we ask everyone to share the job equally and try to make it as easy and low friction as possible. The pre-publish review system is purely editorial review, aimed at helping authors prepare for publishing. The refereeing system is post-publish, it happens through the voting.
eesmith 4y ago
Looks similar to the failed project https://en.wikipedia.org/wiki/Peerage_of_Science https://en.wikipedia.org/wiki/Peerage_of_Science . > Here other scholars who have enough reputation in any of the fields you tagged the paper with can see the draft and offer feedback on it. Speaking for myself, I don't trawl through preprint servers looking for papers to review. As it is, editors have a hard time getting people to review, and even after agreeing to review, to get the review back in reasonable time. > This system gives scholars an enormous amount of control over who they solicit feedback from. How does it handle peer review rings, where authors collude with reviewers to get bogus positive reviews? This is already an issue now. > It gives reviewers an incentive to give solid, constructive review feedback - and rewards good reviewers for their efforts with recognition of their contributions. It offers reviewers an incentive to write a review that the paper author likes. I expect if reviewer writes "this paper duplicates work by Penzias & Wilson (1965) and offers nothing new - reject" then the author will reject it. While "Fix the typo on page 4, otherwise, go ahead and publish" will be accepted - even if the paper really is a waste of time. It's a lot easier to get surface-level proofreading responses than substantive domain-specific responses. > Down voters will be strongly encouraged to post a response explaining their downvote. Why aren't up voters strongly encouraged to post a response explaining their upvote? Why should upvotes not be equally visible? What's the negative consequence should someone not provide a response? > It is not a pre-print server. It is an attempt to replace the journal system with something open to its core, scholar lead, and collectively managed by the scholar community. Any reason to not develop this as an overlay journal on top of existing pre-print servers? Thus reducing a lot of the costs? Why would anyone use this system given PubPeer, and PeerJ, and eLife, and overlay journals, and Publons, and all of the other attempts at breaking the current system, why would someone choose this one?
- dbingham 4y ago
  Hmm... well Peerage of Science was an overlay on top of the journals. It was basically a system the journals would pay for to mediate their own review. Maybe similar tech-wise, but very different goals. > Speaking for myself, I don't trawl through preprint servers looking for papers to review. > As it is, editors have a hard time getting people to review, and even after agreeing to review, to get the review back in reasonable time. Could we change that culture? The hope would be that we could build a culture of people helping each other review, for the simple reason that they want that help themselves when it comes time to for them to publish. With the reputation being more of a bonus. Keep in mind, review here is purely editorial feedback. It is not refereeing - that happens post publish through the voting system. > How does it handle peer review rings, where authors collude with reviewers to get bogus positive reviews? This is already an issue now. See above - review is not refereeing. It could be an issue in refereeing in the votes for sure though. The hope is that the overall weight of the community would override those rings. And in the long term we might even be able to identify them through analysis of the database. If we see the same group of people always up voting each other's down voted papers, we can flag that and take corrective action. > It offers reviewers an incentive to write a review that the paper author likes. I expect if reviewer writes "this paper duplicates work by Penzias & Wilson (1965) and offers nothing new - reject" then the author will reject it. While "Fix the typo on page 4, otherwise, go ahead and publish" will be accepted - even if the paper really is a waste of time. > It's a lot easier to get surface-level proofreading responses than substantive domain-specific responses. That may turn out to be true, but the thought is that the post-publish refereeing system will help incentivize people to take critical feedback seriously at the review step. After all, your goal is to publish something that will be well received. If you get good constructive critical feedback and ignore it, you risk being downvoted to oblivion and losing reputation. > Why aren't up voters strongly encouraged to post a response explaining their upvote? Why should upvotes not be equally visible? > What's the negative consequence should someone not provide a response? It's been asked before and it's definitely on the table. It's something I'd love more feedback on - should responses be required for all votes? Just down votes? Or (as I'm defaulting to) should we follow the StackExchange model of just encouraging responses on down votes? Seems maybe you would argue in favor of requiring responses on all votes? What's the reasoning? > Any reason to not develop this as an overlay journal on top of existing pre-print servers? Thus reducing a lot of the costs? We don't want it to be an overlay, because we want to ultimately replace the system. But it could make a lot of sense to work with the existing pre-print servers. I see no reason why we couldn't start as a pre-print and gradually evolve towards being the final publish, as long as we're clear about the end goal so that we don't just become "yet another pre-print". > Why would anyone use this system given PubPeer, and PeerJ, and eLife, and overlay journals, and Publons, and all of the other attempts at breaking the current system, why would someone choose this one? - PubPeer is just a comment overlay, can't read the papers (as I understand it). - PeerJ is still using the manual model, hence overhead and fees to publish ($1200) rather than diamond open access. This creates equity in access to publishing issues ( a lot of people cannot afford those fees ). - Elife, similarly, charged $3000 per article. These fees create massive equity in access to publishing issues. - Publons I wasn't previously familiar with, but it seems to be about just highlighting review work done for the journals, and not anything to do with open access. None of the attempts to break the current system (that I've found) seem to really be able to replace the journals by doing (almost) everything the journals do. At least, not with out just being a journal and facing the overhead problems. The one thing this project doesn't do that the journals do is curate reviews/refereeing, but the idea is that crowdsourcing could actually work better in the long run - though it's definitely going to bring its own set of problems that will need to be dealt with. But thanks! I hadn't heard of Peerage of Science, and Publons sounds vaguely familiar but I wasn't really familiar with it. I'm constantly learning more about what other efforts have come before and currently exist (there are a lot of them). And thinking through all the questions and counter points is always very helpful!
dougmccune 4y ago
You might want to dig into what ResearchHub is up to https://www.researchhub.com/ https://www.researchhub.com/ If you ignore the web3/crypto aspect, they’ve got a decent start on what seems like a reasonably similar platform. They do all their dev and community building out in the open, so you can join their Discord and listen in on their community calls, etc. Might at least give you some ideas on what to target or avoid.

ISL 4y ago

Consider finding an academic niche in which to really make this product sing; a community that is receptive to the idea. If the PoC can be shown to work in one community, neighboring fields can then point at it and say, "see, this thing is working over in community X. Journal Y is going to take three months to get this reviewed, and this paper is kind of a throwaway -- let's try X and see how it goes." arXiv started in a narrow niche that absolutely needed it, perhaps the same will happen here. Long-term, I'm not sure how this system will pay for itself. Keep costs low and dig in for the long haul. Academia can have very low switching costs (a professor can try your service on a whim), but has enough inertia that it can take decades to turn the ship (deans need to know a prospective hire's impact factor in top-tier journals). Also, as a referee -- pay referees what their time is worth. For real. You'll get real good referees if you do.

Test0129 4y ago
The other side of it is reputation. Academic publishing to a questionable journal (even if it's a good journal) can be a black mark on your career. You not only have to get over the hump of at least break even profit but also convince tenured professors stuck in their ways that your journal will gain and keep a positive reputation. This task is insurmountable for the most part. At least with the current way academia works. Current journals have been around for decades, sometimes even reaching the 100 year mark. Back in the ye olde days it was easy for anyone to publish anything (even non-academics). These days these journals are so deeply entrenched in the minds of academics it would be like convincing 10% of fortune 500 companies to switch from excel to your new cool software. Even it's it better they won't because of this mental block.
- dbingham 4y ago
  Yeah, that may be. But it seems worth trying given all the negative impacts of the status quo.
  - Test0129 4y ago
    I'm not saying to _not_ try it. It's just going to be a dumpster-fire-money-pit that has an extremely high probability of either bankrupting you or producing nothing of significant value. This isn't a shot at OP. The best way to force change here isn't yet another journal, but rather fix the access issues of current journals, convince them it's a good idea to be less exclusive (ever noticed certain initials tend to be published more than others?), and moreover stop the university-journal death-bond that currently is in the market. Good luck at any rate.
- blep_ 4y ago
  Could the solution be "don't actually tie it to the journal"? Have the journal treat itself more as an implementation detail than as an end in itself; you didn't publish a paper in the Journal of Lasers, you just published a paper about lasers. This is still a mental shift for academics to get over, but I wonder if it's a more feasible one.
dbingham 4y ago
> Consider finding an academic niche in which to really make this product sing; a community that is receptive to the idea. If the PoC can be shown to work in one community, neighboring fields can then point at it and say, "see, this thing is working over in community X. Journal Y is going to take three months to get this reviewed, and this paper is kind of a throwaway -- let's try X and see how it goes." > arXiv started in a narrow niche that absolutely needed it, perhaps the same will happen here. That's very similar to my thinking - I'm actually hoping to start with the file drawer problem. When we're ready for open beta, put out a call for file drawered papers. It's also been suggested that we start as pre-print, while being clear about the ultimate intentions to become a journal, and go from there. Maybe even reach out to the existing pre-print servers to partner with them in some way. > Long-term, I'm not sure how this system will pay for itself. Keep costs low and dig in for the long haul. Academia can have very low switching costs (a professor can try your service on a whim), but has enough inertia that it can take decades to turn the ship (deans need to know a prospective hire's impact factor in top-tier journals). Noted. The thought is that it could be funded through donations, grants, and eventually by the institutions if it gains enough traction. But yeah, it's becoming clear that grants at the beginning may be the only way. We'll see how much we get in terms of donations once we hit open beta. > Also, as a referee -- pay referees what their time is worth. For real. You'll get real good referees if you do. ...well... the idea is actually to crowd source refereeing and review. And for people to do it because, well, it needs done. Everyone wants good reviews and everyone knows refereeing needs done, so we ask everyone to share the job equally and try to make it as easy and low friction as possible. The pre-publish review system is purely editorial review, aimed at helping authors prepare for publishing. The refereeing system is post-publish, it happens through the voting.

eesmith 4y ago

Looks similar to the failed project https://en.wikipedia.org/wiki/Peerage_of_Science https://en.wikipedia.org/wiki/Peerage_of_Science . > Here other scholars who have enough reputation in any of the fields you tagged the paper with can see the draft and offer feedback on it. Speaking for myself, I don't trawl through preprint servers looking for papers to review. As it is, editors have a hard time getting people to review, and even after agreeing to review, to get the review back in reasonable time. > This system gives scholars an enormous amount of control over who they solicit feedback from. How does it handle peer review rings, where authors collude with reviewers to get bogus positive reviews? This is already an issue now. > It gives reviewers an incentive to give solid, constructive review feedback - and rewards good reviewers for their efforts with recognition of their contributions. It offers reviewers an incentive to write a review that the paper author likes. I expect if reviewer writes "this paper duplicates work by Penzias & Wilson (1965) and offers nothing new - reject" then the author will reject it. While "Fix the typo on page 4, otherwise, go ahead and publish" will be accepted - even if the paper really is a waste of time. It's a lot easier to get surface-level proofreading responses than substantive domain-specific responses. > Down voters will be strongly encouraged to post a response explaining their downvote. Why aren't up voters strongly encouraged to post a response explaining their upvote? Why should upvotes not be equally visible? What's the negative consequence should someone not provide a response? > It is not a pre-print server. It is an attempt to replace the journal system with something open to its core, scholar lead, and collectively managed by the scholar community. Any reason to not develop this as an overlay journal on top of existing pre-print servers? Thus reducing a lot of the costs? Why would anyone use this system given PubPeer, and PeerJ, and eLife, and overlay journals, and Publons, and all of the other attempts at breaking the current system, why would someone choose this one?

dbingham 4y ago
Hmm... well Peerage of Science was an overlay on top of the journals. It was basically a system the journals would pay for to mediate their own review. Maybe similar tech-wise, but very different goals. > Speaking for myself, I don't trawl through preprint servers looking for papers to review. > As it is, editors have a hard time getting people to review, and even after agreeing to review, to get the review back in reasonable time. Could we change that culture? The hope would be that we could build a culture of people helping each other review, for the simple reason that they want that help themselves when it comes time to for them to publish. With the reputation being more of a bonus. Keep in mind, review here is purely editorial feedback. It is not refereeing - that happens post publish through the voting system. > How does it handle peer review rings, where authors collude with reviewers to get bogus positive reviews? This is already an issue now. See above - review is not refereeing. It could be an issue in refereeing in the votes for sure though. The hope is that the overall weight of the community would override those rings. And in the long term we might even be able to identify them through analysis of the database. If we see the same group of people always up voting each other's down voted papers, we can flag that and take corrective action. > It offers reviewers an incentive to write a review that the paper author likes. I expect if reviewer writes "this paper duplicates work by Penzias & Wilson (1965) and offers nothing new - reject" then the author will reject it. While "Fix the typo on page 4, otherwise, go ahead and publish" will be accepted - even if the paper really is a waste of time. > It's a lot easier to get surface-level proofreading responses than substantive domain-specific responses. That may turn out to be true, but the thought is that the post-publish refereeing system will help incentivize people to take critical feedback seriously at the review step. After all, your goal is to publish something that will be well received. If you get good constructive critical feedback and ignore it, you risk being downvoted to oblivion and losing reputation. > Why aren't up voters strongly encouraged to post a response explaining their upvote? Why should upvotes not be equally visible? > What's the negative consequence should someone not provide a response? It's been asked before and it's definitely on the table. It's something I'd love more feedback on - should responses be required for all votes? Just down votes? Or (as I'm defaulting to) should we follow the StackExchange model of just encouraging responses on down votes? Seems maybe you would argue in favor of requiring responses on all votes? What's the reasoning? > Any reason to not develop this as an overlay journal on top of existing pre-print servers? Thus reducing a lot of the costs? We don't want it to be an overlay, because we want to ultimately replace the system. But it could make a lot of sense to work with the existing pre-print servers. I see no reason why we couldn't start as a pre-print and gradually evolve towards being the final publish, as long as we're clear about the end goal so that we don't just become "yet another pre-print". > Why would anyone use this system given PubPeer, and PeerJ, and eLife, and overlay journals, and Publons, and all of the other attempts at breaking the current system, why would someone choose this one? - PubPeer is just a comment overlay, can't read the papers (as I understand it). - PeerJ is still using the manual model, hence overhead and fees to publish ($1200) rather than diamond open access. This creates equity in access to publishing issues ( a lot of people cannot afford those fees ). - Elife, similarly, charged $3000 per article. These fees create massive equity in access to publishing issues. - Publons I wasn't previously familiar with, but it seems to be about just highlighting review work done for the journals, and not anything to do with open access. None of the attempts to break the current system (that I've found) seem to really be able to replace the journals by doing (almost) everything the journals do. At least, not with out just being a journal and facing the overhead problems. The one thing this project doesn't do that the journals do is curate reviews/refereeing, but the idea is that crowdsourcing could actually work better in the long run - though it's definitely going to bring its own set of problems that will need to be dealt with. But thanks! I hadn't heard of Peerage of Science, and Publons sounds vaguely familiar but I wasn't really familiar with it. I'm constantly learning more about what other efforts have come before and currently exist (there are a lot of them). And thinking through all the questions and counter points is always very helpful!

dougmccune 4y ago

You might want to dig into what ResearchHub is up to https://www.researchhub.com/ https://www.researchhub.com/ If you ignore the web3/crypto aspect, they’ve got a decent start on what seems like a reasonably similar platform. They do all their dev and community building out in the open, so you can join their Discord and listen in on their community calls, etc. Might at least give you some ideas on what to target or avoid.