8 ms·

A Possible Fix for Scientific Publishing

69 points by dbingham 4y ago

dbingham 4y ago
Hey HN, I've had this idea for an open scientific web platform for years. It seemed like such an obvious thing to try. I kept waiting for someone else to build it, but to my knowledge no one else has. I finally saved up enough to take some time away from work and build a proof of concept. I'm looking for feedback on the PoC from scientists, scholars, researchers and academics to try to figure out if there is merit to this idea. I would deeply appreciate any and all feedback, and help getting this request in front of more of the target audience.
- ISL 4y ago
  Consider finding an academic niche in which to really make this product sing; a community that is receptive to the idea. If the PoC can be shown to work in one community, neighboring fields can then point at it and say, "see, this thing is working over in community X. Journal Y is going to take three months to get this reviewed, and this paper is kind of a throwaway -- let's try X and see how it goes." arXiv started in a narrow niche that absolutely needed it, perhaps the same will happen here. Long-term, I'm not sure how this system will pay for itself. Keep costs low and dig in for the long haul. Academia can have very low switching costs (a professor can try your service on a whim), but has enough inertia that it can take decades to turn the ship (deans need to know a prospective hire's impact factor in top-tier journals). Also, as a referee -- pay referees what their time is worth. For real. You'll get real good referees if you do.
  - Test0129 4y ago
    The other side of it is reputation. Academic publishing to a questionable journal (even if it's a good journal) can be a black mark on your career. You not only have to get over the hump of at least break even profit but also convince tenured professors stuck in their ways that your journal will gain and keep a positive reputation. This task is insurmountable for the most part. At least with the current way academia works. Current journals have been around for decades, sometimes even reaching the 100 year mark. Back in the ye olde days it was easy for anyone to publish anything (even non-academics). These days these journals are so deeply entrenched in the minds of academics it would be like convincing 10% of fortune 500 companies to switch from excel to your new cool software. Even it's it better they won't because of this mental block.
    - dbingham 4y ago
      Yeah, that may be. But it seems worth trying given all the negative impacts of the status quo.
      - Test0129 4y ago
        I'm not saying to _not_ try it. It's just going to be a dumpster-fire-money-pit that has an extremely high probability of either bankrupting you or producing nothing of significant value. This isn't a shot at OP. The best way to force change here isn't yet another journal, but rather fix the access issues of current journals, convince them it's a good idea to be less exclusive (ever noticed certain initials tend to be published more than others?), and moreover stop the university-journal death-bond that currently is in the market. Good luck at any rate.
    - blep_ 4y ago
      Could the solution be "don't actually tie it to the journal"? Have the journal treat itself more as an implementation detail than as an end in itself; you didn't publish a paper in the Journal of Lasers, you just published a paper about lasers. This is still a mental shift for academics to get over, but I wonder if it's a more feasible one.
  - dbingham 4y ago
    > Consider finding an academic niche in which to really make this product sing; a community that is receptive to the idea. If the PoC can be shown to work in one community, neighboring fields can then point at it and say, "see, this thing is working over in community X. Journal Y is going to take three months to get this reviewed, and this paper is kind of a throwaway -- let's try X and see how it goes." > arXiv started in a narrow niche that absolutely needed it, perhaps the same will happen here. That's very similar to my thinking - I'm actually hoping to start with the file drawer problem. When we're ready for open beta, put out a call for file drawered papers. It's also been suggested that we start as pre-print, while being clear about the ultimate intentions to become a journal, and go from there. Maybe even reach out to the existing pre-print servers to partner with them in some way. > Long-term, I'm not sure how this system will pay for itself. Keep costs low and dig in for the long haul. Academia can have very low switching costs (a professor can try your service on a whim), but has enough inertia that it can take decades to turn the ship (deans need to know a prospective hire's impact factor in top-tier journals). Noted. The thought is that it could be funded through donations, grants, and eventually by the institutions if it gains enough traction. But yeah, it's becoming clear that grants at the beginning may be the only way. We'll see how much we get in terms of donations once we hit open beta. > Also, as a referee -- pay referees what their time is worth. For real. You'll get real good referees if you do. ...well... the idea is actually to crowd source refereeing and review. And for people to do it because, well, it needs done. Everyone wants good reviews and everyone knows refereeing needs done, so we ask everyone to share the job equally and try to make it as easy and low friction as possible. The pre-publish review system is purely editorial review, aimed at helping authors prepare for publishing. The refereeing system is post-publish, it happens through the voting.
- eesmith 4y ago
  Looks similar to the failed project https://en.wikipedia.org/wiki/Peerage_of_Science https://en.wikipedia.org/wiki/Peerage_of_Science . > Here other scholars who have enough reputation in any of the fields you tagged the paper with can see the draft and offer feedback on it. Speaking for myself, I don't trawl through preprint servers looking for papers to review. As it is, editors have a hard time getting people to review, and even after agreeing to review, to get the review back in reasonable time. > This system gives scholars an enormous amount of control over who they solicit feedback from. How does it handle peer review rings, where authors collude with reviewers to get bogus positive reviews? This is already an issue now. > It gives reviewers an incentive to give solid, constructive review feedback - and rewards good reviewers for their efforts with recognition of their contributions. It offers reviewers an incentive to write a review that the paper author likes. I expect if reviewer writes "this paper duplicates work by Penzias & Wilson (1965) and offers nothing new - reject" then the author will reject it. While "Fix the typo on page 4, otherwise, go ahead and publish" will be accepted - even if the paper really is a waste of time. It's a lot easier to get surface-level proofreading responses than substantive domain-specific responses. > Down voters will be strongly encouraged to post a response explaining their downvote. Why aren't up voters strongly encouraged to post a response explaining their upvote? Why should upvotes not be equally visible? What's the negative consequence should someone not provide a response? > It is not a pre-print server. It is an attempt to replace the journal system with something open to its core, scholar lead, and collectively managed by the scholar community. Any reason to not develop this as an overlay journal on top of existing pre-print servers? Thus reducing a lot of the costs? Why would anyone use this system given PubPeer, and PeerJ, and eLife, and overlay journals, and Publons, and all of the other attempts at breaking the current system, why would someone choose this one?
  - dbingham 4y ago
    Hmm... well Peerage of Science was an overlay on top of the journals. It was basically a system the journals would pay for to mediate their own review. Maybe similar tech-wise, but very different goals. > Speaking for myself, I don't trawl through preprint servers looking for papers to review. > As it is, editors have a hard time getting people to review, and even after agreeing to review, to get the review back in reasonable time. Could we change that culture? The hope would be that we could build a culture of people helping each other review, for the simple reason that they want that help themselves when it comes time to for them to publish. With the reputation being more of a bonus. Keep in mind, review here is purely editorial feedback. It is not refereeing - that happens post publish through the voting system. > How does it handle peer review rings, where authors collude with reviewers to get bogus positive reviews? This is already an issue now. See above - review is not refereeing. It could be an issue in refereeing in the votes for sure though. The hope is that the overall weight of the community would override those rings. And in the long term we might even be able to identify them through analysis of the database. If we see the same group of people always up voting each other's down voted papers, we can flag that and take corrective action. > It offers reviewers an incentive to write a review that the paper author likes. I expect if reviewer writes "this paper duplicates work by Penzias & Wilson (1965) and offers nothing new - reject" then the author will reject it. While "Fix the typo on page 4, otherwise, go ahead and publish" will be accepted - even if the paper really is a waste of time. > It's a lot easier to get surface-level proofreading responses than substantive domain-specific responses. That may turn out to be true, but the thought is that the post-publish refereeing system will help incentivize people to take critical feedback seriously at the review step. After all, your goal is to publish something that will be well received. If you get good constructive critical feedback and ignore it, you risk being downvoted to oblivion and losing reputation. > Why aren't up voters strongly encouraged to post a response explaining their upvote? Why should upvotes not be equally visible? > What's the negative consequence should someone not provide a response? It's been asked before and it's definitely on the table. It's something I'd love more feedback on - should responses be required for all votes? Just down votes? Or (as I'm defaulting to) should we follow the StackExchange model of just encouraging responses on down votes? Seems maybe you would argue in favor of requiring responses on all votes? What's the reasoning? > Any reason to not develop this as an overlay journal on top of existing pre-print servers? Thus reducing a lot of the costs? We don't want it to be an overlay, because we want to ultimately replace the system. But it could make a lot of sense to work with the existing pre-print servers. I see no reason why we couldn't start as a pre-print and gradually evolve towards being the final publish, as long as we're clear about the end goal so that we don't just become "yet another pre-print". > Why would anyone use this system given PubPeer, and PeerJ, and eLife, and overlay journals, and Publons, and all of the other attempts at breaking the current system, why would someone choose this one? - PubPeer is just a comment overlay, can't read the papers (as I understand it). - PeerJ is still using the manual model, hence overhead and fees to publish ($1200) rather than diamond open access. This creates equity in access to publishing issues ( a lot of people cannot afford those fees ). - Elife, similarly, charged $3000 per article. These fees create massive equity in access to publishing issues. - Publons I wasn't previously familiar with, but it seems to be about just highlighting review work done for the journals, and not anything to do with open access. None of the attempts to break the current system (that I've found) seem to really be able to replace the journals by doing (almost) everything the journals do. At least, not with out just being a journal and facing the overhead problems. The one thing this project doesn't do that the journals do is curate reviews/refereeing, but the idea is that crowdsourcing could actually work better in the long run - though it's definitely going to bring its own set of problems that will need to be dealt with. But thanks! I hadn't heard of Peerage of Science, and Publons sounds vaguely familiar but I wasn't really familiar with it. I'm constantly learning more about what other efforts have come before and currently exist (there are a lot of them). And thinking through all the questions and counter points is always very helpful!
- dougmccune 4y ago
  You might want to dig into what ResearchHub is up to https://www.researchhub.com/ https://www.researchhub.com/ If you ignore the web3/crypto aspect, they’ve got a decent start on what seems like a reasonably similar platform. They do all their dev and community building out in the open, so you can join their Discord and listen in on their community calls, etc. Might at least give you some ideas on what to target or avoid.
ta123456789 4y ago
"The ultimate effect of pay-to-play is that the traditional peer review and refereeing process has broken down. If a dishonest researcher gets rejected from a reputable journal, they can take their paper to a pay-to-play journal and have it published there. With over 10,000 academic journals, it’s impossible for the lay public to track which journals are reputable and which are not. As far as the public is concerned, a published paper is a valid paper." Ok, but I take some issue with this. Most scientific papers are not intended at the general public, but at experts working in the specific area that the paper covers. Also, often these papers are not even understandable by a phd outside the area, let alone a lay person. Which journals and authors are reputable are known to experts in the area. "Pay to play" papers published in obscure journals will not get much attention or be taken seriously.
- throwaway1777 4y ago
  You are correct they shouldnt be intended for public to take seriously. I think one issue is that the news media frequently spins these articles so the public ends up needing to decipher the accuracy.
- Test0129 4y ago
  Agree. When I was in graduate school the "bad journals" were well known. The pay to play journals are very cleverly disguised, but it's extremely hard to accidentally publish to or cite from one of these journals. There are many, many problems with the almost cult-like atmosphere of reviewers in good journals but pay-to-play isn't one of them.
- dbingham 4y ago
  So here's the experience driving that observation: I'm adjacent to a number of anti-vax communities and during the pandemic (and before) there were PDFs of "1000 papers showing why vaccines are dangerous" being passed around. Honestly, snake oil salesmen don't even have to go through pay to play, they can just self publish and the effect is the same. If it looks real enough, a lot of people don't have the background to tell the difference. They look at the abstract and the conclusions and go from there. If we're going to beat that kind of misinformation, the real stuff needs to be open and available so that translators can help people understand it. There were people in academia doing this on social media during the pandemic, but we need many, many more of them. There are a lot of people outside of academia who have enough of a background that they could help - if only they had access. Edit to add: Also, I've heard from some folks in academia that they've been bitten by open access journals that started out well intentioned and turned into pay to play with bad reputations. And that in some fields, it is hard to track which ones are good and which ones are going bad.
periram 4y ago
Awesome effort. I really really hope this makes at least a minor dent in the scientific publishing deadlock. My 2 cents: Please be open to suggestions. Please involve more people, who you feel comfortable working with. You have solved multiple independent problems of decentralized trust, voting etc. Please try to be agnostic to a specific solution.
- dbingham 4y ago
  Will do my best! That's part of the reasoning behind early ask for feedback and my eagerness for it :) Trying to get people involved as early in the process as possible to get more perspectives.
sseagull 4y ago
A few comments from a practicing scientist 1.) You have a lot of nice tech, but think long and hard about the social dynamics and how you can get from A->B. By far, the tech will be the easy part. Convincing established scientists and up-and-coming scientists who need publications in quality journals is much, much harder. As another comment says, try finding a niche first and build out. Particularly some less established fields. 2.) You mention hiring engineers, designers, devops, QE. This points to solving a social problem with technology. Consider marketing, or even social scientists who can guide you in understanding the communities you are attempting to influence. Become embedded in those communities, build trust, attend (and maybe sponsor) conferences, and then branch out. 3.) Sort of a pet peeve of mine (among many): You talk about making the studies accessible to the lay public (in terms of understandability). I feel this is a bad trend. Scientists need a way to talk to each other in their own language, without needing to lay out all the context every time. Academic publishing is that avenue. I have published many articles that will not be understandable to anyone without at least 2 years of graduate school in the field. They aren't groundbreaking, but can still be important. Science is very incremental in many areas. Although I 100% agree that it should be open and free. For the lay public, there are always books, blogs, and pop-sci news that put it into context. Anyway, I wish you success (truly, we need improvement over the status quo). It's easy for people to criticize, but think of all the things we don't criticize (which means we approve of them!)
- dbingham 4y ago
  Thanks for this feedback - it's very helpful! 1) Yep. This I know. The tech is the easy part, by far. Any suggestions on which fields might be most open to this approach, and where to approach them? 2) Good point. At the moment, there is a lot of technical work to do, so that's where my head is. But if we get big enough to where we can afford it, community moderators who can help guide the community towards a collaborative approach and gently nudge back some of the egos I've already been told (many times over) are a problem were already on the list. Social scientists are a great suggestion that I had not thought of. 3) This may be an area where we disagree - at least partially. I understand the need to talk to peers in a mutual language, and I'm not arguing against that. My thought is that this platform could help provide space for translators. Papers and responses would be in the technical language of the field, but I've thought about including a (heavily moderated) peanut gallery below for translators to help the lay public understand the work. One of the issues that has been front and center in the pandemic is that a lot of bad vaccine science was easily accessible to the public, and when it was done intentionally by snake oil salesmen, intentionally made easily understandable. But the good stuff was locked up behind paywalls or hard to understand. There were translators on social media who did their best to answer the concerns but there weren't enough of them. There are a lot of people not currently in academia who have enough of a foundation to tell snake oil from the real deal and who could help translate, with access. Right now, they don't have access. So you wind up with PDFs of "1000 papers showing why vaccines are harmful" getting shared around and no easy way for people adjacent to the communities where they are being shared to counter them. Additionally, science is supposed to be an open process - we're supposed to trust it not because of appeals to authority, but because, in theory, we can replicate the reasoning and experiments that got us to our current understanding. Now, of course, replicating the experiments is mostly impossible for the lay person at this point, but someone who's interested and has the time should be able to follow along. We can't say, in the same breath, "You should trust science, because it's open and you can replicate it" and "Oh, you can't understand this." We need to create space for scientists to talk to each other, and for translators to help the public understand and follow the conversation in real time. We would all benefit from a greater understanding of all aspects of science, and from the increased trust that come with understanding.
jpmattia 4y ago
As an observation, I think "If you're not paying for it, you're the product" applies here. In traditional publishing, the people writing the papers are the product. Journals compete to get the best papers, and those that got the best papers in the immediate past tend to get the best paper submissions in the future. In open-access publishing, the readers are the product. Having good papers is only one way to build your product base, On the other side, there are a number of ways to market your product (pay-to-play, for example.) While I like open-access in principle, I also like the filtering from the traditional pubs to help me narrow a reading list.
- dbingham 4y ago
  How would you apply that observation to Wikipedia? That's the business model we're aiming to take - crowdsourced, crowdfunded, non-profit, and community governed. And the reason for that is precisely because of the mis-aligned incentives in both the pay to access and pay to publish models. The filtering here would be accomplished by the tagging system and the voting system.
  - jpmattia 4y ago
    > How would you apply that observation to Wikipedia? I would notice that the Wikipedia reader base is a multiple of peer-reviewed scientific-paper base. While I don't know what the multiple is, I would guess the range is 1-2 orders of magnitude. Wikipedia claims its average donation is $15 [1], so a rough guess will imply $100-$1000 average donation for a sci pub model. That's on the income side. On the spending side: I would guess the major difference in costs would be the scientific editors, which exist in the traditional science publication model. There are probably large initial marketing costs associated with 1. finding the right target authors and convincing them to send a valuable contribution to the new site, 2. finding serious editors to manage peer review and separate the wheat from the chaff. (edit: I assume this is still part of the process. I think about something like Einstein's special relativity paper, and I'd have to guess it would have been widely downvoted as an internet crank trying to publish crazy ideas -- he did not have a reputation yet.) I see this as potentially expensive: The level of the first papers will determine how likely it is to succeed. Said differently: If the new site starts by publishing lame work, it's not like someone is going to want to publish there afterward. In any case: I think it is a tough and worthwhile problem to solve, and I wish you good luck. [1] https://donate.wikimedia.org/wiki/FAQ https://donate.wikimedia.org/wiki/FAQ
CJefferson 4y ago
I find the idea that papers are judged by a simple up-vote/down-vote terrifying. Popular academics will post their new paper on Twitter, which will immediately lead to a round of upvotes from their friends and followers. People who are currently out of favour will be downvote-bombed. I see no suggested way of stopping this -- and I hope noone thinks the answer is academics are "above that". I'd suggest changing from "people may leave a comment if they wish when they downvote" to "any vote requires at least 200 words of justification". In general I think it's important reviewing requires thought and justification.
- dbingham 4y ago
  This is helpful feedback, and requiring responses with votes was already something we were pondering. And I'm increasingly leaning in that direction. One thing that wasn't covered in the blog post was the intention for downvotes to cost reputation, same as on SE right now. That should help disincentivize brigading to some degree. The field system may also help counter that tendency to some degree. By choosing which fields you publish in, you can choose the subset of people able to review and vote. But there is also a degree to which we will try to counter this with culture. The cultures of internet communities can be extremely stick and resistant to change, both good and bad. Just think about Reddit vs StackExchange. Roughly similar systems (though with very different goals and guidelines), very different cultures. If we can establish a strong culture on the platform from the get go - a culture of low ego collaboration and seeking good science above all else, that can potentially help counter some of the worse tendencies of wider academia as more and more of it moves into the platform. At least in the early days, before we can afford active moderation. In the long run, active moderation, and tools for community moderation, can help as well.
Kydlaw 4y ago
Having left academia after my PhD, but still doing research in the private sector, I hate the scientific publishing industry wholeheartedly. It is something of the past that is definitely keeping us, as a society, from making progress. There are more and more initiatives to create new tools around the literature (Semantic Scholar, ResearchRabbit and stuff like that) but less around moving he publishing industry as a whole. (I'm not affiliated with the previous organizations though I think Semantic Scholar have an API for peer review matching that might be useful in your case). If you need contributors I'd be interested.
- tensor 4y ago
  I don't think scientific publishing is perfect, but I strongly disagree that it's somehow holding back society. T he number one thing that is holding back society in my opinion is that the vast majority don't think science is important or useful, and even worse, think that their opinions are somehow the same as scientific knowledge. Even convincing people in industry that basing decisions on data can be hard, let alone convincing them to use something approximating the scientific method. In short, if people wanted to read science papers but couldn't, I might agree with you, but people don't even want to.
deoxyribo 4y ago
I love what you’re trying to accomplish here! I encourage you to experiment aggressively and try to focus on testing the biggest risks, rather than prioritizing the easiest things to make progress on but whose success or failure won’t impact your overall odds of achieving your goal. One thing which I’ve noticed in your response to other comments is that your plan for controlling bad behavior by participants in your system often boils down to either having a strong culture means no one will behave poorly, or that voting and crowd consensus is an effective system for both motivating good behavior and punishing bad behavior. This probably works fine as long as (1) the community is very small such that everyone actually knows everyone else, or (2) there isn’t much actually at stake. You reference Stack Overflow and Wikipedia - these are examples where not much is at stake. If accumulating points and reputation in either of these systems would determine who gets tenure and the culmination of lifelong career ambitions, then I wonder if those systems would be as robust to bad actors. A white-hot risk I think you should focus on to experiment is that peer-reviewed academic science is not a friendly collegial system but an aggressive and high-stakes game where people have very very big incentive to game the system. The current system has tons of problems, but it does function in the face of participants who would like to cheat if they could. The big influential editors and reviewers act like a leviathan in the Hobbesian sense which causes many problems but also solves many. I encourage you to think about how you could solve some of these potential issues that the current system doesn’t have so much, particularly around preventing various forms of bad behavior which are easily caught and punished in a more hierarchical system like the current one, but which a more crowd-sourced system might fall prey to. Things like collusion rings to upvote each others’ papers, using lay popularity or scientific fame to overwhelm legitimate complaints about issues with a paper, even bot nets or paying to win by farming upvotes - all things which current journals and peer review are basically immune to. People will try this with their careers and livelihoods on the line in a way Wikipedia editors or Stack Overflow users just wouldn’t care to. These issues can be overcome, but for what it’s worth I think this is the hardest part of what you’re trying to accomplish, and more important for you to focus on than the tech stack or getting a first few papers published on your setup. It will be hard to simulate, but I think you should focus on how to test the behavior of a scientist who is desperate and has their back against the wall in the publish or perish game, and how you can design to reward good behavior and prevent bad behavior in the worst case, not just in the best case or even the average case. Very cool effort. If you pull this off the positive externalities could be huge.
deleted 4y ago
[deleted]
t_mann 4y ago
Awesome idea! Only half-joking: something like the HN ranking algorithm could be great for that, maybe you can reach out to them if they can help (you might even try applying for YC). Some feedback: I think the division by disciplines needs to be stronger than tags - researchers are so specialized that most research even within the same discipline is often unintelligible for those specialized in a different sub-area. They need to know roughly what to expect when they go to the platform, you don't want to show physics papers to sociologists. Think of the preprint servers, they're also constrained to some disciplines (arXiv for mathematical disciplines, bioRxiv and medRxiv, SSRN for economics/social sciences...). In the bigger picture, I think moving towards such a model would be a good thing also for scientific debate in the public eyes. Going from the binary 'It's published' to 'It seems most experts in this area tend to agree with this paper's conclusions, with some important caveats' would give a much better picture of the state of our knowledge at the cutting edge, and perhaps also address the bias against negative results better than journals who pledged to do so can.
istillwritecode 4y ago
Reviewing is not the cause of the economic problems in academic publishing. The economic problems arise from the car at of copy and production editing. That's where the money goes. See https://f1000research.com/articles/10-20 https://f1000research.com/articles/10-20
- joshjob42 4y ago
  I think the idea is that almost all of the costs like management, lobbying, paywalls, etc. are to be done away with, which that article estimates is at least 50% of the cost. Getting rid of a profit margin by being a public non-profit org slashes another 30%, and so presumably a system like this may only cost a few hundred dollars an article on average, well below the thousands of dollars typically charged for top journals. At that point it's easy for costs to be handled by libraries and institutions contributing to operating costs at much less than (the author of the OP claims <1% but I'd guess more like ~20%) the cost of existing journal access and publishing fees paid by institutions today.
tgv 4y ago
I don't get the positive reactions. This sounds like: let's do to scientific publishing what social media have done to society. And that's make it worse. For fistful of dollars. - Academics advance based on your publication score. They will avoid your platform if it doesn't help them get grants or tenure. If they've got it, they will avoid it even more. - Only a few people will put time in reviewing and handing out points, where unfortunately reviewing means: three seconds of scanning and gut feeling. You won't be able to keep even the most blatantly fraudulent papers out. - There will be a few good articles that score well, and you'll defend your platform by pointing that out, but below that there will be a mass of highly rated papers that don't deserve it. They'll be there because a group with an agenda put them there, or because groups actively keep papers of "competitors" down. - As said by others: the "average citizen" isn't part of this. Never has been, and doesn't need to. Open Access is for fellow scientists and very specialized citizens. - "With over 10,000 academic journals, it’s impossible for the lay public to track which journals are reputable and which are not." That's not true. Most researchers only look in a small number of journals and can distinguish wheat from chaff. But you won't have 10k journals: you're aiming at millions of reviewers. - Why use voting by god-knows-whom if you can use the same "trust" mechanism that's used today: citations? Because you'll only become a record keeper. There won't be a need to fund your system.
murphyslab 4y ago
> The platform splits pre-publish peer review from post-publish refereeing. Pre-publish review then becomes completely about helping authors polish their work and decide if their articles are ready to publish. Refereeing happens post-publish, and in a way which is easily understandable to the lay reader, helping the general public sort solid studies from shakey ones. The author doesn't seem to understand the "referee" role of (associate) editors and reviewers. The core idea is about gatekeeping of a journal title's self-set publication limit in favour of articles that represent a substantive advancement of a field or concept. However this itself is a problem for many fields of research, since it's a core factor underlying the failure to engage in replication studies and to check if results are valid. Moreover this would tend to encourage researchers with small but useful results to shelve them and NOT publish, since it will create a negative feedback mechanism for such contributions: > Up votes increase the paper’s score and grant the authors reputation in the tagged fields. Down votes decrease the paper’s score and the author’s reputation in the tagged fields. Part of this problem is because, like any site which uses an upvote or downvote function, the button will be used to express things other than the intended meaning. "Does not contribute" quickly becomes "I don't like X" (even if it is methodologically valid).
Gatsky 4y ago
My experience of papers that went to top tier journals is that the peer reviews were detailed, interesting and clearly from people who knew what they were talking about. They were also somewhat harsh and demanding. It is almost certainly the case that these reviewers would not expend the same effort on a platform like this. They review papers in Nature etc becuase they feel that they are participating in something important, at the cutting edge of science. They also, somewhere along the line, feel that doing such reviews will improve their chances of publishing there. Additionally Nature will offer some reviewers the chance to write an editorial about the article they reviewed. These are strong incentives that successfully coerce busy people. How will you replicate this? What can happen with these open peer review platforms is that they attract 'hyperreviewers' of dubious quality. This can be seen with publons. Look at the top ranked peer reviewer there: https://publons.com/wos-op/researcher/1217819/kaveh-ostad-ali-askari/metrics/ https://publons.com/wos-op/researcher/1217819/kaveh-ostad-al..., who apparently reviewed 1352 articles in 2021. Perhaps this person is actually carefully reviewing 5 articles every working day of the year... then again perhaps not.
dbingham 4y ago
So, please know that I am I reading all the critical comments very carefully and taking them seriously. I haven't responded to all of them, partly because I don't have good answers to all the criticisms and I need to sit with them for a few days. I have also gotten a lot of positive feedback and encouragement. I'd say it's 50/50 right now. Maybe 60/40 at best. I've got a lot of thinking to do. I think what it comes down to is this: Do you believe the average actor in academia/science is a good actor? In other words, are they honestly trying hard to collaboratively advance good work and good science and check their own biases? Or do you believe the average actor is a bad actor (OR that the bad incentive structure is so strong that it corrupts even the well intentioned into mediocre actors at best.) If the former is true (average actor is a good actor), then this system could function the way it is supposed to and would be self regulating to a large degree. And there are many things we could build into it that would help that. If the latter is true, then the critics are right and this system will absolutely fail. Catastrophically. I really don't want to believe the average academic actor is a bad actor, so I'm going to forge ahead for now. But I'm also going to be seeking a lot more feedback for sure. I'm going to read every single critical comment, note it, spend time thinking about the ways in which this could go bad that have been detailed here (some of which I was aware of and thinking about, and some of which are novel ideas). And if I see this heading in those directions, and I can't turn it back on track, I will absolutely pull the plug rather than keep pushing a platform doing harm.
drpixie 4y ago
I believe all scientific publishing should be open access, but it doesn't address the fundamental problem with scientific publishing - there just just too much of it! Due to the well known pressures of academia, and of CV writers everywhere, we live in a world where many people (apparently) work on hundreds or thousands of papers; in a world of "minimum publishable work"; in a world where every Masters and PhD is expected to have published. And the vast majority of those papers are almost pointless, low quality, and/or almost content free! (I admit my own publications into that majority.) There is no way that any human is genuinely across all the literature of their field. And the situation is getting worse and worse, rapidly. I don't know how we'd do it, but perhaps we could move towards a situation where we have (say) "papers" and "notes" - where "papers" are very high quality and published only when something big and important comes up, and where "notes" are used to record everything else. Someone new to the field, or looking for anything important, need be familiar with (only) a few hundred papers; and all those "new technique", "minor variation", "new species", "confirmation", and "we did stuff" publications would be "notes". Of course, this would depend on a truly scathing response when anyone puts forward low quality "papers". PS. It was once that case that publications divided along similar lines as journal papers and conference papers. No longer seems so much the case.
deleted 4y ago
[deleted]
j-pb 4y ago
Theres an issue with this "one review fits all" approach. Scientific communities are often splintered into sub-communities with different views and allegiances. The scientific community is not a set, it's a graph with different connectedness and weights. A imho better system would: * use public key crypto to give everybody a pseudonym under which they can create, sign and publish articles * a publication graph that is used both as a reference system and a review system; - a reference is a link from one article to another if it reuses and builds upon it (standing on the shoulder of giants) - a review is a link from one article to another that passes judgement on or reproduces that article (peer review) * a sentiment analysis based system that turns the review links into peer-review quality metrics * a way to delegate judgement from yourself to others that you trust in your community * a page-rank like algorithm that computes trust relative to your private key and the trust it has assigned/delegated * and most importantly: a UI that shows you a papers ranking and relevance based on YOUR personal trust graph and community
toomim 4y ago
I'm also working in this area; and your work is impressive! I find your thinking to be very clear, and thank you very much for taking the time to articulate this space so clearly. Your idea is diligently thought-through, with admirable execution. I see a limitation in your design — it relies on a central agent to design and ultimately adjudicate its reputation system. That is not safe for science! But worse, scientists will be hesitant to buy into and rely on a system that depends on such centralized leadership. I'd encourage you to compare those aspects of your design with the decentralized "Subjective Reputation System" that we are building: https://peeryview.org/about https://peeryview.org/about Peery View's ratings are Subjective: each user votes on not just publications, but also other users, to express who he trusts, and can even define his own filter function to generate his own news feed of high-ranked publications. And as each user votes, his votes are shared with everyone else who votes him up, and so on, which creates a trust network that can cover the entire web. Peery View is a protocol, instead of a platform. Every user can post his votes and publications on any web server he chooses. The Braid Protocol extensions to HTTP make it trivial for his votes and feeds to synchronize with other user's votes on their servers. This means we don't need a central server, and we don't need a 501c3, and we don't need to set the culture. The culture will evolve itself, because every user has an incentive to have a good feed, and thus has an incentive to vote on good posts and good user behavior, and then he also has an incentive to share these votes, because doing so will give him reputation in the scientific community around him, as other people vote him up, because doing so will improve their feeds. You find that in this way, the system has built-in resistance to being gamed. If any bias or manipulation can be systematically detected (e.g. a sybill attack) by any user, that user can express compensating votes that undo the bias and penalize the manipulation. This user can then express those compensating votes, and get reputation in the network, as other users vote him up to get improved feeds. Thus emerges a distributed incentive for the community to root out untruths. As the community deliberates the best ways to find truth, they articulate the scientific principles that they find work the best. This is a very healthy, and much-needed dialogue for us to be having these days, with the crisis in scientific culture that we are all experiencing.

69 points by dbingham 4y ago

Hey HN, I've had this idea for an open scientific web platform for years. It seemed like such an obvious thing to try. I kept waiting for someone else to build it, but to my knowledge no one else has. I finally saved up enough to take some time away from work and build a proof of concept. I'm looking for feedback on the PoC from scientists, scholars, researchers and academics to try to figure out if there is merit to this idea. I would deeply appreciate any and all feedback, and help getting this request in front of more of the target audience.

ISL 4y ago
Consider finding an academic niche in which to really make this product sing; a community that is receptive to the idea. If the PoC can be shown to work in one community, neighboring fields can then point at it and say, "see, this thing is working over in community X. Journal Y is going to take three months to get this reviewed, and this paper is kind of a throwaway -- let's try X and see how it goes." arXiv started in a narrow niche that absolutely needed it, perhaps the same will happen here. Long-term, I'm not sure how this system will pay for itself. Keep costs low and dig in for the long haul. Academia can have very low switching costs (a professor can try your service on a whim), but has enough inertia that it can take decades to turn the ship (deans need to know a prospective hire's impact factor in top-tier journals). Also, as a referee -- pay referees what their time is worth. For real. You'll get real good referees if you do.
- Test0129 4y ago
  The other side of it is reputation. Academic publishing to a questionable journal (even if it's a good journal) can be a black mark on your career. You not only have to get over the hump of at least break even profit but also convince tenured professors stuck in their ways that your journal will gain and keep a positive reputation. This task is insurmountable for the most part. At least with the current way academia works. Current journals have been around for decades, sometimes even reaching the 100 year mark. Back in the ye olde days it was easy for anyone to publish anything (even non-academics). These days these journals are so deeply entrenched in the minds of academics it would be like convincing 10% of fortune 500 companies to switch from excel to your new cool software. Even it's it better they won't because of this mental block.
  - dbingham 4y ago
    Yeah, that may be. But it seems worth trying given all the negative impacts of the status quo.
    - Test0129 4y ago
      I'm not saying to _not_ try it. It's just going to be a dumpster-fire-money-pit that has an extremely high probability of either bankrupting you or producing nothing of significant value. This isn't a shot at OP. The best way to force change here isn't yet another journal, but rather fix the access issues of current journals, convince them it's a good idea to be less exclusive (ever noticed certain initials tend to be published more than others?), and moreover stop the university-journal death-bond that currently is in the market. Good luck at any rate.
  - blep_ 4y ago
    Could the solution be "don't actually tie it to the journal"? Have the journal treat itself more as an implementation detail than as an end in itself; you didn't publish a paper in the Journal of Lasers, you just published a paper about lasers. This is still a mental shift for academics to get over, but I wonder if it's a more feasible one.
- dbingham 4y ago
  > Consider finding an academic niche in which to really make this product sing; a community that is receptive to the idea. If the PoC can be shown to work in one community, neighboring fields can then point at it and say, "see, this thing is working over in community X. Journal Y is going to take three months to get this reviewed, and this paper is kind of a throwaway -- let's try X and see how it goes." > arXiv started in a narrow niche that absolutely needed it, perhaps the same will happen here. That's very similar to my thinking - I'm actually hoping to start with the file drawer problem. When we're ready for open beta, put out a call for file drawered papers. It's also been suggested that we start as pre-print, while being clear about the ultimate intentions to become a journal, and go from there. Maybe even reach out to the existing pre-print servers to partner with them in some way. > Long-term, I'm not sure how this system will pay for itself. Keep costs low and dig in for the long haul. Academia can have very low switching costs (a professor can try your service on a whim), but has enough inertia that it can take decades to turn the ship (deans need to know a prospective hire's impact factor in top-tier journals). Noted. The thought is that it could be funded through donations, grants, and eventually by the institutions if it gains enough traction. But yeah, it's becoming clear that grants at the beginning may be the only way. We'll see how much we get in terms of donations once we hit open beta. > Also, as a referee -- pay referees what their time is worth. For real. You'll get real good referees if you do. ...well... the idea is actually to crowd source refereeing and review. And for people to do it because, well, it needs done. Everyone wants good reviews and everyone knows refereeing needs done, so we ask everyone to share the job equally and try to make it as easy and low friction as possible. The pre-publish review system is purely editorial review, aimed at helping authors prepare for publishing. The refereeing system is post-publish, it happens through the voting.
eesmith 4y ago
Looks similar to the failed project https://en.wikipedia.org/wiki/Peerage_of_Science https://en.wikipedia.org/wiki/Peerage_of_Science . > Here other scholars who have enough reputation in any of the fields you tagged the paper with can see the draft and offer feedback on it. Speaking for myself, I don't trawl through preprint servers looking for papers to review. As it is, editors have a hard time getting people to review, and even after agreeing to review, to get the review back in reasonable time. > This system gives scholars an enormous amount of control over who they solicit feedback from. How does it handle peer review rings, where authors collude with reviewers to get bogus positive reviews? This is already an issue now. > It gives reviewers an incentive to give solid, constructive review feedback - and rewards good reviewers for their efforts with recognition of their contributions. It offers reviewers an incentive to write a review that the paper author likes. I expect if reviewer writes "this paper duplicates work by Penzias & Wilson (1965) and offers nothing new - reject" then the author will reject it. While "Fix the typo on page 4, otherwise, go ahead and publish" will be accepted - even if the paper really is a waste of time. It's a lot easier to get surface-level proofreading responses than substantive domain-specific responses. > Down voters will be strongly encouraged to post a response explaining their downvote. Why aren't up voters strongly encouraged to post a response explaining their upvote? Why should upvotes not be equally visible? What's the negative consequence should someone not provide a response? > It is not a pre-print server. It is an attempt to replace the journal system with something open to its core, scholar lead, and collectively managed by the scholar community. Any reason to not develop this as an overlay journal on top of existing pre-print servers? Thus reducing a lot of the costs? Why would anyone use this system given PubPeer, and PeerJ, and eLife, and overlay journals, and Publons, and all of the other attempts at breaking the current system, why would someone choose this one?
- dbingham 4y ago
  Hmm... well Peerage of Science was an overlay on top of the journals. It was basically a system the journals would pay for to mediate their own review. Maybe similar tech-wise, but very different goals. > Speaking for myself, I don't trawl through preprint servers looking for papers to review. > As it is, editors have a hard time getting people to review, and even after agreeing to review, to get the review back in reasonable time. Could we change that culture? The hope would be that we could build a culture of people helping each other review, for the simple reason that they want that help themselves when it comes time to for them to publish. With the reputation being more of a bonus. Keep in mind, review here is purely editorial feedback. It is not refereeing - that happens post publish through the voting system. > How does it handle peer review rings, where authors collude with reviewers to get bogus positive reviews? This is already an issue now. See above - review is not refereeing. It could be an issue in refereeing in the votes for sure though. The hope is that the overall weight of the community would override those rings. And in the long term we might even be able to identify them through analysis of the database. If we see the same group of people always up voting each other's down voted papers, we can flag that and take corrective action. > It offers reviewers an incentive to write a review that the paper author likes. I expect if reviewer writes "this paper duplicates work by Penzias & Wilson (1965) and offers nothing new - reject" then the author will reject it. While "Fix the typo on page 4, otherwise, go ahead and publish" will be accepted - even if the paper really is a waste of time. > It's a lot easier to get surface-level proofreading responses than substantive domain-specific responses. That may turn out to be true, but the thought is that the post-publish refereeing system will help incentivize people to take critical feedback seriously at the review step. After all, your goal is to publish something that will be well received. If you get good constructive critical feedback and ignore it, you risk being downvoted to oblivion and losing reputation. > Why aren't up voters strongly encouraged to post a response explaining their upvote? Why should upvotes not be equally visible? > What's the negative consequence should someone not provide a response? It's been asked before and it's definitely on the table. It's something I'd love more feedback on - should responses be required for all votes? Just down votes? Or (as I'm defaulting to) should we follow the StackExchange model of just encouraging responses on down votes? Seems maybe you would argue in favor of requiring responses on all votes? What's the reasoning? > Any reason to not develop this as an overlay journal on top of existing pre-print servers? Thus reducing a lot of the costs? We don't want it to be an overlay, because we want to ultimately replace the system. But it could make a lot of sense to work with the existing pre-print servers. I see no reason why we couldn't start as a pre-print and gradually evolve towards being the final publish, as long as we're clear about the end goal so that we don't just become "yet another pre-print". > Why would anyone use this system given PubPeer, and PeerJ, and eLife, and overlay journals, and Publons, and all of the other attempts at breaking the current system, why would someone choose this one? - PubPeer is just a comment overlay, can't read the papers (as I understand it). - PeerJ is still using the manual model, hence overhead and fees to publish ($1200) rather than diamond open access. This creates equity in access to publishing issues ( a lot of people cannot afford those fees ). - Elife, similarly, charged $3000 per article. These fees create massive equity in access to publishing issues. - Publons I wasn't previously familiar with, but it seems to be about just highlighting review work done for the journals, and not anything to do with open access. None of the attempts to break the current system (that I've found) seem to really be able to replace the journals by doing (almost) everything the journals do. At least, not with out just being a journal and facing the overhead problems. The one thing this project doesn't do that the journals do is curate reviews/refereeing, but the idea is that crowdsourcing could actually work better in the long run - though it's definitely going to bring its own set of problems that will need to be dealt with. But thanks! I hadn't heard of Peerage of Science, and Publons sounds vaguely familiar but I wasn't really familiar with it. I'm constantly learning more about what other efforts have come before and currently exist (there are a lot of them). And thinking through all the questions and counter points is always very helpful!
dougmccune 4y ago
You might want to dig into what ResearchHub is up to https://www.researchhub.com/ https://www.researchhub.com/ If you ignore the web3/crypto aspect, they’ve got a decent start on what seems like a reasonably similar platform. They do all their dev and community building out in the open, so you can join their Discord and listen in on their community calls, etc. Might at least give you some ideas on what to target or avoid.

ta123456789 4y ago

"The ultimate effect of pay-to-play is that the traditional peer review and refereeing process has broken down. If a dishonest researcher gets rejected from a reputable journal, they can take their paper to a pay-to-play journal and have it published there. With over 10,000 academic journals, it’s impossible for the lay public to track which journals are reputable and which are not. As far as the public is concerned, a published paper is a valid paper." Ok, but I take some issue with this. Most scientific papers are not intended at the general public, but at experts working in the specific area that the paper covers. Also, often these papers are not even understandable by a phd outside the area, let alone a lay person. Which journals and authors are reputable are known to experts in the area. "Pay to play" papers published in obscure journals will not get much attention or be taken seriously.

throwaway1777 4y ago
You are correct they shouldnt be intended for public to take seriously. I think one issue is that the news media frequently spins these articles so the public ends up needing to decipher the accuracy.
Test0129 4y ago
Agree. When I was in graduate school the "bad journals" were well known. The pay to play journals are very cleverly disguised, but it's extremely hard to accidentally publish to or cite from one of these journals. There are many, many problems with the almost cult-like atmosphere of reviewers in good journals but pay-to-play isn't one of them.
dbingham 4y ago
So here's the experience driving that observation: I'm adjacent to a number of anti-vax communities and during the pandemic (and before) there were PDFs of "1000 papers showing why vaccines are dangerous" being passed around. Honestly, snake oil salesmen don't even have to go through pay to play, they can just self publish and the effect is the same. If it looks real enough, a lot of people don't have the background to tell the difference. They look at the abstract and the conclusions and go from there. If we're going to beat that kind of misinformation, the real stuff needs to be open and available so that translators can help people understand it. There were people in academia doing this on social media during the pandemic, but we need many, many more of them. There are a lot of people outside of academia who have enough of a background that they could help - if only they had access. Edit to add: Also, I've heard from some folks in academia that they've been bitten by open access journals that started out well intentioned and turned into pay to play with bad reputations. And that in some fields, it is hard to track which ones are good and which ones are going bad.

periram 4y ago

Awesome effort. I really really hope this makes at least a minor dent in the scientific publishing deadlock. My 2 cents: Please be open to suggestions. Please involve more people, who you feel comfortable working with. You have solved multiple independent problems of decentralized trust, voting etc. Please try to be agnostic to a specific solution.

dbingham 4y ago
Will do my best! That's part of the reasoning behind early ask for feedback and my eagerness for it :) Trying to get people involved as early in the process as possible to get more perspectives.

sseagull 4y ago

A few comments from a practicing scientist 1.) You have a lot of nice tech, but think long and hard about the social dynamics and how you can get from A->B. By far, the tech will be the easy part. Convincing established scientists and up-and-coming scientists who need publications in quality journals is much, much harder. As another comment says, try finding a niche first and build out. Particularly some less established fields. 2.) You mention hiring engineers, designers, devops, QE. This points to solving a social problem with technology. Consider marketing, or even social scientists who can guide you in understanding the communities you are attempting to influence. Become embedded in those communities, build trust, attend (and maybe sponsor) conferences, and then branch out. 3.) Sort of a pet peeve of mine (among many): You talk about making the studies accessible to the lay public (in terms of understandability). I feel this is a bad trend. Scientists need a way to talk to each other in their own language, without needing to lay out all the context every time. Academic publishing is that avenue. I have published many articles that will not be understandable to anyone without at least 2 years of graduate school in the field. They aren't groundbreaking, but can still be important. Science is very incremental in many areas. Although I 100% agree that it should be open and free. For the lay public, there are always books, blogs, and pop-sci news that put it into context. Anyway, I wish you success (truly, we need improvement over the status quo). It's easy for people to criticize, but think of all the things we don't criticize (which means we approve of them!)

dbingham 4y ago
Thanks for this feedback - it's very helpful! 1) Yep. This I know. The tech is the easy part, by far. Any suggestions on which fields might be most open to this approach, and where to approach them? 2) Good point. At the moment, there is a lot of technical work to do, so that's where my head is. But if we get big enough to where we can afford it, community moderators who can help guide the community towards a collaborative approach and gently nudge back some of the egos I've already been told (many times over) are a problem were already on the list. Social scientists are a great suggestion that I had not thought of. 3) This may be an area where we disagree - at least partially. I understand the need to talk to peers in a mutual language, and I'm not arguing against that. My thought is that this platform could help provide space for translators. Papers and responses would be in the technical language of the field, but I've thought about including a (heavily moderated) peanut gallery below for translators to help the lay public understand the work. One of the issues that has been front and center in the pandemic is that a lot of bad vaccine science was easily accessible to the public, and when it was done intentionally by snake oil salesmen, intentionally made easily understandable. But the good stuff was locked up behind paywalls or hard to understand. There were translators on social media who did their best to answer the concerns but there weren't enough of them. There are a lot of people not currently in academia who have enough of a foundation to tell snake oil from the real deal and who could help translate, with access. Right now, they don't have access. So you wind up with PDFs of "1000 papers showing why vaccines are harmful" getting shared around and no easy way for people adjacent to the communities where they are being shared to counter them. Additionally, science is supposed to be an open process - we're supposed to trust it not because of appeals to authority, but because, in theory, we can replicate the reasoning and experiments that got us to our current understanding. Now, of course, replicating the experiments is mostly impossible for the lay person at this point, but someone who's interested and has the time should be able to follow along. We can't say, in the same breath, "You should trust science, because it's open and you can replicate it" and "Oh, you can't understand this." We need to create space for scientists to talk to each other, and for translators to help the public understand and follow the conversation in real time. We would all benefit from a greater understanding of all aspects of science, and from the increased trust that come with understanding.

jpmattia 4y ago

As an observation, I think "If you're not paying for it, you're the product" applies here. In traditional publishing, the people writing the papers are the product. Journals compete to get the best papers, and those that got the best papers in the immediate past tend to get the best paper submissions in the future. In open-access publishing, the readers are the product. Having good papers is only one way to build your product base, On the other side, there are a number of ways to market your product (pay-to-play, for example.) While I like open-access in principle, I also like the filtering from the traditional pubs to help me narrow a reading list.

dbingham 4y ago
How would you apply that observation to Wikipedia? That's the business model we're aiming to take - crowdsourced, crowdfunded, non-profit, and community governed. And the reason for that is precisely because of the mis-aligned incentives in both the pay to access and pay to publish models. The filtering here would be accomplished by the tagging system and the voting system.
- jpmattia 4y ago
  > How would you apply that observation to Wikipedia? I would notice that the Wikipedia reader base is a multiple of peer-reviewed scientific-paper base. While I don't know what the multiple is, I would guess the range is 1-2 orders of magnitude. Wikipedia claims its average donation is $15 [1], so a rough guess will imply $100-$1000 average donation for a sci pub model. That's on the income side. On the spending side: I would guess the major difference in costs would be the scientific editors, which exist in the traditional science publication model. There are probably large initial marketing costs associated with 1. finding the right target authors and convincing them to send a valuable contribution to the new site, 2. finding serious editors to manage peer review and separate the wheat from the chaff. (edit: I assume this is still part of the process. I think about something like Einstein's special relativity paper, and I'd have to guess it would have been widely downvoted as an internet crank trying to publish crazy ideas -- he did not have a reputation yet.) I see this as potentially expensive: The level of the first papers will determine how likely it is to succeed. Said differently: If the new site starts by publishing lame work, it's not like someone is going to want to publish there afterward. In any case: I think it is a tough and worthwhile problem to solve, and I wish you good luck. [1] https://donate.wikimedia.org/wiki/FAQ https://donate.wikimedia.org/wiki/FAQ

CJefferson 4y ago

I find the idea that papers are judged by a simple up-vote/down-vote terrifying. Popular academics will post their new paper on Twitter, which will immediately lead to a round of upvotes from their friends and followers. People who are currently out of favour will be downvote-bombed. I see no suggested way of stopping this -- and I hope noone thinks the answer is academics are "above that". I'd suggest changing from "people may leave a comment if they wish when they downvote" to "any vote requires at least 200 words of justification". In general I think it's important reviewing requires thought and justification.

dbingham 4y ago
This is helpful feedback, and requiring responses with votes was already something we were pondering. And I'm increasingly leaning in that direction. One thing that wasn't covered in the blog post was the intention for downvotes to cost reputation, same as on SE right now. That should help disincentivize brigading to some degree. The field system may also help counter that tendency to some degree. By choosing which fields you publish in, you can choose the subset of people able to review and vote. But there is also a degree to which we will try to counter this with culture. The cultures of internet communities can be extremely stick and resistant to change, both good and bad. Just think about Reddit vs StackExchange. Roughly similar systems (though with very different goals and guidelines), very different cultures. If we can establish a strong culture on the platform from the get go - a culture of low ego collaboration and seeking good science above all else, that can potentially help counter some of the worse tendencies of wider academia as more and more of it moves into the platform. At least in the early days, before we can afford active moderation. In the long run, active moderation, and tools for community moderation, can help as well.

Kydlaw 4y ago

Having left academia after my PhD, but still doing research in the private sector, I hate the scientific publishing industry wholeheartedly. It is something of the past that is definitely keeping us, as a society, from making progress. There are more and more initiatives to create new tools around the literature (Semantic Scholar, ResearchRabbit and stuff like that) but less around moving he publishing industry as a whole. (I'm not affiliated with the previous organizations though I think Semantic Scholar have an API for peer review matching that might be useful in your case). If you need contributors I'd be interested.

tensor 4y ago
I don't think scientific publishing is perfect, but I strongly disagree that it's somehow holding back society. T he number one thing that is holding back society in my opinion is that the vast majority don't think science is important or useful, and even worse, think that their opinions are somehow the same as scientific knowledge. Even convincing people in industry that basing decisions on data can be hard, let alone convincing them to use something approximating the scientific method. In short, if people wanted to read science papers but couldn't, I might agree with you, but people don't even want to.

deoxyribo 4y ago

I love what you’re trying to accomplish here! I encourage you to experiment aggressively and try to focus on testing the biggest risks, rather than prioritizing the easiest things to make progress on but whose success or failure won’t impact your overall odds of achieving your goal. One thing which I’ve noticed in your response to other comments is that your plan for controlling bad behavior by participants in your system often boils down to either having a strong culture means no one will behave poorly, or that voting and crowd consensus is an effective system for both motivating good behavior and punishing bad behavior. This probably works fine as long as (1) the community is very small such that everyone actually knows everyone else, or (2) there isn’t much actually at stake. You reference Stack Overflow and Wikipedia - these are examples where not much is at stake. If accumulating points and reputation in either of these systems would determine who gets tenure and the culmination of lifelong career ambitions, then I wonder if those systems would be as robust to bad actors. A white-hot risk I think you should focus on to experiment is that peer-reviewed academic science is not a friendly collegial system but an aggressive and high-stakes game where people have very very big incentive to game the system. The current system has tons of problems, but it does function in the face of participants who would like to cheat if they could. The big influential editors and reviewers act like a leviathan in the Hobbesian sense which causes many problems but also solves many. I encourage you to think about how you could solve some of these potential issues that the current system doesn’t have so much, particularly around preventing various forms of bad behavior which are easily caught and punished in a more hierarchical system like the current one, but which a more crowd-sourced system might fall prey to. Things like collusion rings to upvote each others’ papers, using lay popularity or scientific fame to overwhelm legitimate complaints about issues with a paper, even bot nets or paying to win by farming upvotes - all things which current journals and peer review are basically immune to. People will try this with their careers and livelihoods on the line in a way Wikipedia editors or Stack Overflow users just wouldn’t care to. These issues can be overcome, but for what it’s worth I think this is the hardest part of what you’re trying to accomplish, and more important for you to focus on than the tech stack or getting a first few papers published on your setup. It will be hard to simulate, but I think you should focus on how to test the behavior of a scientist who is desperate and has their back against the wall in the publish or perish game, and how you can design to reward good behavior and prevent bad behavior in the worst case, not just in the best case or even the average case. Very cool effort. If you pull this off the positive externalities could be huge.

deleted 4y ago

[deleted]

t_mann 4y ago

Awesome idea! Only half-joking: something like the HN ranking algorithm could be great for that, maybe you can reach out to them if they can help (you might even try applying for YC). Some feedback: I think the division by disciplines needs to be stronger than tags - researchers are so specialized that most research even within the same discipline is often unintelligible for those specialized in a different sub-area. They need to know roughly what to expect when they go to the platform, you don't want to show physics papers to sociologists. Think of the preprint servers, they're also constrained to some disciplines (arXiv for mathematical disciplines, bioRxiv and medRxiv, SSRN for economics/social sciences...). In the bigger picture, I think moving towards such a model would be a good thing also for scientific debate in the public eyes. Going from the binary 'It's published' to 'It seems most experts in this area tend to agree with this paper's conclusions, with some important caveats' would give a much better picture of the state of our knowledge at the cutting edge, and perhaps also address the bias against negative results better than journals who pledged to do so can.

istillwritecode 4y ago

Reviewing is not the cause of the economic problems in academic publishing. The economic problems arise from the car at of copy and production editing. That's where the money goes. See https://f1000research.com/articles/10-20 https://f1000research.com/articles/10-20

joshjob42 4y ago
I think the idea is that almost all of the costs like management, lobbying, paywalls, etc. are to be done away with, which that article estimates is at least 50% of the cost. Getting rid of a profit margin by being a public non-profit org slashes another 30%, and so presumably a system like this may only cost a few hundred dollars an article on average, well below the thousands of dollars typically charged for top journals. At that point it's easy for costs to be handled by libraries and institutions contributing to operating costs at much less than (the author of the OP claims <1% but I'd guess more like ~20%) the cost of existing journal access and publishing fees paid by institutions today.

tgv 4y ago

I don't get the positive reactions. This sounds like: let's do to scientific publishing what social media have done to society. And that's make it worse. For fistful of dollars. - Academics advance based on your publication score. They will avoid your platform if it doesn't help them get grants or tenure. If they've got it, they will avoid it even more. - Only a few people will put time in reviewing and handing out points, where unfortunately reviewing means: three seconds of scanning and gut feeling. You won't be able to keep even the most blatantly fraudulent papers out. - There will be a few good articles that score well, and you'll defend your platform by pointing that out, but below that there will be a mass of highly rated papers that don't deserve it. They'll be there because a group with an agenda put them there, or because groups actively keep papers of "competitors" down. - As said by others: the "average citizen" isn't part of this. Never has been, and doesn't need to. Open Access is for fellow scientists and very specialized citizens. - "With over 10,000 academic journals, it’s impossible for the lay public to track which journals are reputable and which are not." That's not true. Most researchers only look in a small number of journals and can distinguish wheat from chaff. But you won't have 10k journals: you're aiming at millions of reviewers. - Why use voting by god-knows-whom if you can use the same "trust" mechanism that's used today: citations? Because you'll only become a record keeper. There won't be a need to fund your system.

murphyslab 4y ago

> The platform splits pre-publish peer review from post-publish refereeing. Pre-publish review then becomes completely about helping authors polish their work and decide if their articles are ready to publish. Refereeing happens post-publish, and in a way which is easily understandable to the lay reader, helping the general public sort solid studies from shakey ones. The author doesn't seem to understand the "referee" role of (associate) editors and reviewers. The core idea is about gatekeeping of a journal title's self-set publication limit in favour of articles that represent a substantive advancement of a field or concept. However this itself is a problem for many fields of research, since it's a core factor underlying the failure to engage in replication studies and to check if results are valid. Moreover this would tend to encourage researchers with small but useful results to shelve them and NOT publish, since it will create a negative feedback mechanism for such contributions: > Up votes increase the paper’s score and grant the authors reputation in the tagged fields. Down votes decrease the paper’s score and the author’s reputation in the tagged fields. Part of this problem is because, like any site which uses an upvote or downvote function, the button will be used to express things other than the intended meaning. "Does not contribute" quickly becomes "I don't like X" (even if it is methodologically valid).

Gatsky 4y ago

My experience of papers that went to top tier journals is that the peer reviews were detailed, interesting and clearly from people who knew what they were talking about. They were also somewhat harsh and demanding. It is almost certainly the case that these reviewers would not expend the same effort on a platform like this. They review papers in Nature etc becuase they feel that they are participating in something important, at the cutting edge of science. They also, somewhere along the line, feel that doing such reviews will improve their chances of publishing there. Additionally Nature will offer some reviewers the chance to write an editorial about the article they reviewed. These are strong incentives that successfully coerce busy people. How will you replicate this? What can happen with these open peer review platforms is that they attract 'hyperreviewers' of dubious quality. This can be seen with publons. Look at the top ranked peer reviewer there: https://publons.com/wos-op/researcher/1217819/kaveh-ostad-ali-askari/metrics/ https://publons.com/wos-op/researcher/1217819/kaveh-ostad-al..., who apparently reviewed 1352 articles in 2021. Perhaps this person is actually carefully reviewing 5 articles every working day of the year... then again perhaps not.

dbingham 4y ago

So, please know that I am I reading all the critical comments very carefully and taking them seriously. I haven't responded to all of them, partly because I don't have good answers to all the criticisms and I need to sit with them for a few days. I have also gotten a lot of positive feedback and encouragement. I'd say it's 50/50 right now. Maybe 60/40 at best. I've got a lot of thinking to do. I think what it comes down to is this: Do you believe the average actor in academia/science is a good actor? In other words, are they honestly trying hard to collaboratively advance good work and good science and check their own biases? Or do you believe the average actor is a bad actor (OR that the bad incentive structure is so strong that it corrupts even the well intentioned into mediocre actors at best.) If the former is true (average actor is a good actor), then this system could function the way it is supposed to and would be self regulating to a large degree. And there are many things we could build into it that would help that. If the latter is true, then the critics are right and this system will absolutely fail. Catastrophically. I really don't want to believe the average academic actor is a bad actor, so I'm going to forge ahead for now. But I'm also going to be seeking a lot more feedback for sure. I'm going to read every single critical comment, note it, spend time thinking about the ways in which this could go bad that have been detailed here (some of which I was aware of and thinking about, and some of which are novel ideas). And if I see this heading in those directions, and I can't turn it back on track, I will absolutely pull the plug rather than keep pushing a platform doing harm.

drpixie 4y ago

I believe all scientific publishing should be open access, but it doesn't address the fundamental problem with scientific publishing - there just just too much of it! Due to the well known pressures of academia, and of CV writers everywhere, we live in a world where many people (apparently) work on hundreds or thousands of papers; in a world of "minimum publishable work"; in a world where every Masters and PhD is expected to have published. And the vast majority of those papers are almost pointless, low quality, and/or almost content free! (I admit my own publications into that majority.) There is no way that any human is genuinely across all the literature of their field. And the situation is getting worse and worse, rapidly. I don't know how we'd do it, but perhaps we could move towards a situation where we have (say) "papers" and "notes" - where "papers" are very high quality and published only when something big and important comes up, and where "notes" are used to record everything else. Someone new to the field, or looking for anything important, need be familiar with (only) a few hundred papers; and all those "new technique", "minor variation", "new species", "confirmation", and "we did stuff" publications would be "notes". Of course, this would depend on a truly scathing response when anyone puts forward low quality "papers". PS. It was once that case that publications divided along similar lines as journal papers and conference papers. No longer seems so much the case.

deleted 4y ago

[deleted]

j-pb 4y ago

Theres an issue with this "one review fits all" approach. Scientific communities are often splintered into sub-communities with different views and allegiances. The scientific community is not a set, it's a graph with different connectedness and weights. A imho better system would: * use public key crypto to give everybody a pseudonym under which they can create, sign and publish articles * a publication graph that is used both as a reference system and a review system; - a reference is a link from one article to another if it reuses and builds upon it (standing on the shoulder of giants) - a review is a link from one article to another that passes judgement on or reproduces that article (peer review) * a sentiment analysis based system that turns the review links into peer-review quality metrics * a way to delegate judgement from yourself to others that you trust in your community * a page-rank like algorithm that computes trust relative to your private key and the trust it has assigned/delegated * and most importantly: a UI that shows you a papers ranking and relevance based on YOUR personal trust graph and community

toomim 4y ago

I'm also working in this area; and your work is impressive! I find your thinking to be very clear, and thank you very much for taking the time to articulate this space so clearly. Your idea is diligently thought-through, with admirable execution. I see a limitation in your design — it relies on a central agent to design and ultimately adjudicate its reputation system. That is not safe for science! But worse, scientists will be hesitant to buy into and rely on a system that depends on such centralized leadership. I'd encourage you to compare those aspects of your design with the decentralized "Subjective Reputation System" that we are building: https://peeryview.org/about https://peeryview.org/about Peery View's ratings are Subjective: each user votes on not just publications, but also other users, to express who he trusts, and can even define his own filter function to generate his own news feed of high-ranked publications. And as each user votes, his votes are shared with everyone else who votes him up, and so on, which creates a trust network that can cover the entire web. Peery View is a protocol, instead of a platform. Every user can post his votes and publications on any web server he chooses. The Braid Protocol extensions to HTTP make it trivial for his votes and feeds to synchronize with other user's votes on their servers. This means we don't need a central server, and we don't need a 501c3, and we don't need to set the culture. The culture will evolve itself, because every user has an incentive to have a good feed, and thus has an incentive to vote on good posts and good user behavior, and then he also has an incentive to share these votes, because doing so will give him reputation in the scientific community around him, as other people vote him up, because doing so will improve their feeds. You find that in this way, the system has built-in resistance to being gamed. If any bias or manipulation can be systematically detected (e.g. a sybill attack) by any user, that user can express compensating votes that undo the bias and penalize the manipulation. This user can then express those compensating votes, and get reputation in the network, as other users vote him up to get improved feeds. Thus emerges a distributed incentive for the community to root out untruths. As the community deliberates the best ways to find truth, they articulate the scientific principles that they find work the best. This is a very healthy, and much-needed dialogue for us to be having these days, with the crisis in scientific culture that we are all experiencing.