4 ms·

Academic and Scientific publishing. It's the primary source material of human knowledge. It should be completely open and accessible to everyone with no barri

Academic and Scientific publishing. It's the primary source material of human knowledge. It should be completely open and accessible to everyone with no barriers to access the literature or to add to the literature. The structure it currently takes - academic journals - made perfect sense in the 1600s when that structure was developed, and it continued to work reasonably well for distributing academic results up into the early 1900s. But then it got privatized. Now there are tens of thousands of academic journals, 80% of them charge a fee for access, and most of the remainder charge a fee to publish. Often thousands of dollars. Given that science works in the aggregate and you can't know if you have the real answer to your question until you've accessed all of the literature on the topic, this structure is now making it impossible for people not in the institution to even figure out what we know on a topic. And hard even for people in the institution. The ultimate decider of policy in a democracy is the average citizen. The people who decide our government policy don't have access to the primary source material of science. In the US, a lot of really important policy is set at the municipal level (city, county), and in that case the people actually writing and implementing the policy also don't have access. If we can develop a web platform that does all the journals do (match papers to qualified reviewers, maintain literature integrity by filtering out bad work, and dole out reputation), then I think it's possible we could draw publishing out of the journals and into the open where it belongs. I've got an idea for a platform that does all that by crowdsourcing it (the journals are already crowdsourcing it - just manually using an editor). It's basically Github+StackExchange for academic publishing. It works by tying the reputation system to academic fields, so papers are tagged with fields and then reputation is gained and lost in those fields. I'm building it now, I'm month or two out from beginning a closed beta. (Aiming for end of October, beginning of November.) I wrote up a detailed description here: https://blog.peer-review.io/we-might-have-a-way-to-fix-scientific-publishing https://blog.peer-review.io/we-might-have-a-way-to-fix-scien...

solardev 4y ago
Important work! Thanks for trying to do something about it.
mirror_neuron 4y ago
If you haven’t heard of it, you may be interested in openreview.net They operate mostly in the computer science academic community, last time I checked.
- dbingham 4y ago
  Yep! I came across them. There are a ton of different projects and platforms out there trying to tackle this problem. But so far every alternative I've come across can only handle some of the various services the journals are providing (or are trying to side step and redesign the whole process from the ground up, which is an even bigger hill to climb). For example, there are several attempts to overlay review on pre-prints and open repositories (like Zenodo), but they aren't identifying qualified reviewers and matching them to papers in any way. There are the repositories and pre-prints themselves, but they aren't providing review. There are attempts to build open journals, but they're still taking the journal form, which means they still have lots of manual overhead. Most of them have varying amounts of traction, but few seem to actually be on a course to replace the whole system. ...it remains to be seen whether peer-review.io will fare any better. I think it has a chance because it does provide an alternative system for each of the services the journals provide. Well -- except one, which is the moderator role the editor sometimes plays. That may prove fatal, but only time will tell.
w10-1 4y ago
As you suggest, journals are a critical area for society. Since we're on HN, we get to take off our technical hat and put on our product hat -- not the fit-to-customer product, but disrupt-the-industry product. What are the incentive structures that would destroy this or make it work? Consider the scenarios today... - What's wrong with the thousands of similar-sounding journals popping up on China publishing research that is either unvetted or copied from other journals, so researchers can satisfy their publish-or-perish needs? - How do you deal with reviewer networks? - How do you get the best of the best, people with no time and plenty of opportunities and money, to provide effective feedback on an avalanche of articles? - How do you get the support of, without being dominated by, the big research companies and universities? - Look at the history of reputation systems, from quora to stackexchange to games, on a matrix with the difficulty of determining correctness of questions: -- Who won the game? -- Does this code work? -- Which library should I use? -- Is this paper correct and valuable? - What about prediction markets, which try to use money/investment as a measure of seriousness: how would that help or hurt? (and isn't that what we're doing by insisting academics publish for tenure?) In the spirit of MVP, you might consider a pivot to a much smaller problem: how to professional groups establish and document their standard of care? A team wiki? (gets stale, disconnected from operations) Run books? (too stepwise to convey meaning needed to transform the system) Slack-capture? email? documentation? What about capturing the benefit in a change of format, from opaque text/pdf to something like a per-domain semantic web, with connections to empirical methods and findings? That could provide incentives for everyone. Like the professional-group scenario, the killer feature would be something that grows incrementally with multiple authors (as code does for developers).
imnotreallynew 4y ago
I’ve been interested in building a similar platform for quite some time. I actually bought the scholar.io domain for that purpose, but never got around to it.
- dbingham 4y ago
  I'm working open source and would welcome contributions! (https://github.com/danielbingham/peerreview https://github.com/danielbingham/peerreview) (Although, the first contribution would probably need to be getting the local working again in a new context... I've been going fast and taking on some techdebt that will need to be paid down soon.)
bo1024 4y ago
Quick suggestion from a researcher: people will only want to submit if there’s an editorial team with high trust and reputation for their specific area. Automated reviewer matching won’t be good enough to find decent reviewers. I would suggest partnering with people in very specific disciplines who want to break away and establish open access journals, but want to focus on the editor and review process rather than the logistics (which you provide).
- bo1024 4y ago
  I would call it something more specific than peer review.
  - dbingham 4y ago
    Name suggestions welcome! I suck at naming - Peer Review was a working title intended to be somewhat tongue in cheek (similar to StackOverflow). But... because I suck at naming, it is, of course, a little on the nose. So far I haven't come up with anything better.
    - Sugimot0 4y ago
      Unless 'peer review' elicits negative feelings in the target audience it seems like a pretty good name. It relates to the core concept, it's respectable and easily memorable. Seems like good branding but I'm no expert, nor am I the target audience. To toss a few names in the hat, off the top of my head: - openresearch - colab - openjournal - stud.io Probably collect a few ideas and run a small survey for the initial audience
      - bo1024 4y ago
        My concern is that "peer review" is a wide term, it would be kind of like calling a startup Web Browser or Network Engineering. Too much confusion.
- dbingham 4y ago
  This is on the roadmap. The plan is to create a "Journal" entity on the site. Editors can create a Journal on the site and then create teams of reviewers which they tag with fields. Authors submitting papers can then submit those papers to community review, or one or more journals, or both. Reviews coming from journal's teams will be highlighted. At any point during the review process, once the journal's team is satisfied, they can mark the paper with their badge of approval. Papers can collect badges from multiple journals. If the authors disagree with the journals team, or just get tired of waiting, they can still publish at any time just like with community review. For authors who choose both, the journal's team and community reviewers can interact with each other just as in normal community review. I think it's a way to provide a stepping stone for people from the existing system towards full community review and would provide logistics to those organizing open journals with teams of high quality reviewers. The plan is to implement it during the closed beta period. I would welcome feedback on this concept as well :)
ss2f 4y ago
This is a brilliant idea! If you think the problem is bad in the US, you haven't stepped into spaces like Asia. This is a path that can help the world, not just EU/US scenarios. May I also suggest you consider the network layer centralization. When you mentioned Github+Stackoverflow, I got the point even before I visited your site. However, even as you think about an alternative on how we publish, consider that technical questions can have significant political consequences. I am of the view that centralized networks are a major contribution to the situation we find ourselves in today. Distributed/Decentralized/Federated options like ActivityPub may help in your journey in what surely is a great idea. Check Lemmy, for example, on a real stackoverflow option - https://join-lemmy.org/ https://join-lemmy.org/ and Gitea already working on a federated "Github".
svnt 4y ago
I love this. Given the other connections to software analogs, any chance you are planning to add plumbing for some sort of citation dependency tracking? Like if a major paper is withdrawn, or a theory disproven, could you get a list of significantly impacted papers?
origin_path 4y ago
"The ultimate decider of policy in a democracy is the average citizen. The people who decide our government policy don't have access to the primary source material of science." Often they do - quite a lot of stuff is open access. The biggest issue with getting people to read scientific papers is the demonization of those who actually do it. A lot of papers are terribly misleading so the moment outsiders start engaging with the literature they conclude a lot of science is junk, and when they act on that they're immediately attacked as "science deniers" ... by the sort of people who don't read papers, because they are convinced they don't need to. Getting more stuff as open access will only accelerate the decline in trust in science, by allowing more people to see what's going on behind the curtain. That is not necessarily an issue, but just so you're aware of that - it isn't going to lead to lots of people suddenly basing policy on scientific papers. It's going to lead to a lot of scientists getting defunded.

solardev 4y ago

Important work! Thanks for trying to do something about it.

mirror_neuron 4y ago

If you haven’t heard of it, you may be interested in openreview.net They operate mostly in the computer science academic community, last time I checked.

dbingham 4y ago
Yep! I came across them. There are a ton of different projects and platforms out there trying to tackle this problem. But so far every alternative I've come across can only handle some of the various services the journals are providing (or are trying to side step and redesign the whole process from the ground up, which is an even bigger hill to climb). For example, there are several attempts to overlay review on pre-prints and open repositories (like Zenodo), but they aren't identifying qualified reviewers and matching them to papers in any way. There are the repositories and pre-prints themselves, but they aren't providing review. There are attempts to build open journals, but they're still taking the journal form, which means they still have lots of manual overhead. Most of them have varying amounts of traction, but few seem to actually be on a course to replace the whole system. ...it remains to be seen whether peer-review.io will fare any better. I think it has a chance because it does provide an alternative system for each of the services the journals provide. Well -- except one, which is the moderator role the editor sometimes plays. That may prove fatal, but only time will tell.

w10-1 4y ago

As you suggest, journals are a critical area for society. Since we're on HN, we get to take off our technical hat and put on our product hat -- not the fit-to-customer product, but disrupt-the-industry product. What are the incentive structures that would destroy this or make it work? Consider the scenarios today... - What's wrong with the thousands of similar-sounding journals popping up on China publishing research that is either unvetted or copied from other journals, so researchers can satisfy their publish-or-perish needs? - How do you deal with reviewer networks? - How do you get the best of the best, people with no time and plenty of opportunities and money, to provide effective feedback on an avalanche of articles? - How do you get the support of, without being dominated by, the big research companies and universities? - Look at the history of reputation systems, from quora to stackexchange to games, on a matrix with the difficulty of determining correctness of questions: -- Who won the game? -- Does this code work? -- Which library should I use? -- Is this paper correct and valuable? - What about prediction markets, which try to use money/investment as a measure of seriousness: how would that help or hurt? (and isn't that what we're doing by insisting academics publish for tenure?) In the spirit of MVP, you might consider a pivot to a much smaller problem: how to professional groups establish and document their standard of care? A team wiki? (gets stale, disconnected from operations) Run books? (too stepwise to convey meaning needed to transform the system) Slack-capture? email? documentation? What about capturing the benefit in a change of format, from opaque text/pdf to something like a per-domain semantic web, with connections to empirical methods and findings? That could provide incentives for everyone. Like the professional-group scenario, the killer feature would be something that grows incrementally with multiple authors (as code does for developers).

imnotreallynew 4y ago

I’ve been interested in building a similar platform for quite some time. I actually bought the scholar.io domain for that purpose, but never got around to it.

dbingham 4y ago
I'm working open source and would welcome contributions! (https://github.com/danielbingham/peerreview https://github.com/danielbingham/peerreview) (Although, the first contribution would probably need to be getting the local working again in a new context... I've been going fast and taking on some techdebt that will need to be paid down soon.)

bo1024 4y ago

Quick suggestion from a researcher: people will only want to submit if there’s an editorial team with high trust and reputation for their specific area. Automated reviewer matching won’t be good enough to find decent reviewers. I would suggest partnering with people in very specific disciplines who want to break away and establish open access journals, but want to focus on the editor and review process rather than the logistics (which you provide).

bo1024 4y ago
I would call it something more specific than peer review.
- dbingham 4y ago
  Name suggestions welcome! I suck at naming - Peer Review was a working title intended to be somewhat tongue in cheek (similar to StackOverflow). But... because I suck at naming, it is, of course, a little on the nose. So far I haven't come up with anything better.
  - Sugimot0 4y ago
    Unless 'peer review' elicits negative feelings in the target audience it seems like a pretty good name. It relates to the core concept, it's respectable and easily memorable. Seems like good branding but I'm no expert, nor am I the target audience. To toss a few names in the hat, off the top of my head: - openresearch - colab - openjournal - stud.io Probably collect a few ideas and run a small survey for the initial audience
    - bo1024 4y ago
      My concern is that "peer review" is a wide term, it would be kind of like calling a startup Web Browser or Network Engineering. Too much confusion.
dbingham 4y ago
This is on the roadmap. The plan is to create a "Journal" entity on the site. Editors can create a Journal on the site and then create teams of reviewers which they tag with fields. Authors submitting papers can then submit those papers to community review, or one or more journals, or both. Reviews coming from journal's teams will be highlighted. At any point during the review process, once the journal's team is satisfied, they can mark the paper with their badge of approval. Papers can collect badges from multiple journals. If the authors disagree with the journals team, or just get tired of waiting, they can still publish at any time just like with community review. For authors who choose both, the journal's team and community reviewers can interact with each other just as in normal community review. I think it's a way to provide a stepping stone for people from the existing system towards full community review and would provide logistics to those organizing open journals with teams of high quality reviewers. The plan is to implement it during the closed beta period. I would welcome feedback on this concept as well :)

ss2f 4y ago

This is a brilliant idea! If you think the problem is bad in the US, you haven't stepped into spaces like Asia. This is a path that can help the world, not just EU/US scenarios. May I also suggest you consider the network layer centralization. When you mentioned Github+Stackoverflow, I got the point even before I visited your site. However, even as you think about an alternative on how we publish, consider that technical questions can have significant political consequences. I am of the view that centralized networks are a major contribution to the situation we find ourselves in today. Distributed/Decentralized/Federated options like ActivityPub may help in your journey in what surely is a great idea. Check Lemmy, for example, on a real stackoverflow option - https://join-lemmy.org/ https://join-lemmy.org/ and Gitea already working on a federated "Github".

svnt 4y ago

I love this. Given the other connections to software analogs, any chance you are planning to add plumbing for some sort of citation dependency tracking? Like if a major paper is withdrawn, or a theory disproven, could you get a list of significantly impacted papers?

origin_path 4y ago

"The ultimate decider of policy in a democracy is the average citizen. The people who decide our government policy don't have access to the primary source material of science." Often they do - quite a lot of stuff is open access. The biggest issue with getting people to read scientific papers is the demonization of those who actually do it. A lot of papers are terribly misleading so the moment outsiders start engaging with the literature they conclude a lot of science is junk, and when they act on that they're immediately attacked as "science deniers" ... by the sort of people who don't read papers, because they are convinced they don't need to. Getting more stuff as open access will only accelerate the decline in trust in science, by allowing more people to see what's going on behind the curtain. That is not necessarily an issue, but just so you're aware of that - it isn't going to lead to lots of people suddenly basing policy on scientific papers. It's going to lead to a lot of scientists getting defunded.