5 ms·
Any ideas for how the system might work differently? Could peer review learn anything from the open source community?
by chrisco255 6y ago
Any ideas for how the system might work differently? Could peer review learn anything from the open source community?
- WoodenChair 6y ago> Could peer review learn anything from the open source community? I think that's a great question. Imagine papers with pull requests, issues, and additional experiments contributed from anyone with relevant experience. And imagine those experiments being open data and up for third-party verification before publication. That could be powerful.
- chrisco255 6y agoYes, and a reputation system built in but still democratic enough that any modern-day "Einstein" could break through.
- Vinnl 6y agoAn article detailing this and similar ideas can be found here, for those interested: https://doi.org/10.12688/f1000research.12037.3 https://doi.org/10.12688/f1000research.12037.3
- pdonis 6y agoThe ostensible purpose of peer review is filtering: making sure that the signal to noise ratio in published journals remains high enough for the journals to be useful to other scientists. That logic probably made sense back when journals were the primary means of communicating scientific results. But the Internet has changed the situation drastically. Scientists don't need to wait for a journal to publish things; they just go to arxiv.org. Nor do they need to wait for a journal to filter things through peer review; they have plenty of other ways of judging papers for themselves. So at this point I don't think we need a "system" at all for what peer review used to accomplish.
- rland 6y agoWhat other ways are you referring to?
- pdonis 6y agoTheir own judgment on reading the paper, talking to other scientists they know, going to conferences, watching or participating in online discussions, etc.
- nine_k 6y agoThis works if your time is infinite.
- DoreenMichele 6y agoNot even then. I used to be on a health list that was ostensibly looking for a "cure" or treatment for cystic fibrosis. They frequently posted links to abstracts of papers and discussed it on list and I had mixed feelings about that. I often didn't understand the medical terminology and it was just a blurb and yadda. At some point, I talked on the phone with a guy with a PhD that I knew through an entirely unrelated email list. And he told me to not bother to read the abstract if I couldn't get hold of the full paper because it often said something different from what the paper said. I was relieved to hear this and not at all surprised, so I quit reading those abstracts and trying to pretend to participate in those discussions or whatever. Some years later, a woman on that same list offered to co-author a paper with me and give me credit and yadda. And then she changed her mind and began trashing me on list and unsubscribed. Her son died of CF (at the age of 17) years and years before and she was in search for some pill to make it all better, obsessed with finding a cure because of his death. I use a lifestyle based approach. She was a smoker. I think the implication of my ideas is that it's partly her fault her son died. And I think her guilt over that possibility is why she's obsessed with finding an answer, but it has to be an answer that somehow absolves her of guilt and doesn't in any way suggest it's her fault. It needs to do the opposite for her to be satisfied. I was homeless at the time that she offered to write a paper with me. Once she kicked me to the curb, I didn't really have any place else to go. Anyway, regardless of your process, you need some signals of some sort. I relied on the opinion of the guy with the PhD in part because he had a PhD. And I think my life has been better for that. It's good to have a so-called "bullshit detector" but one of the ways we filter information is by relying on our understanding that some people can be believed even if we don't understand what they are saying because it is over our head.
- hitekker 6y agoThis paper proposes a return to editorial review: https://sci-hub.tw/https://doi.org/10.1007/s11017-012-9233-1 https://sci-hub.tw/https://doi.org/10.1007/s11017-012-9233-1
- cycomanic 6y agoI think a first step is to make reviews public, in particular also rejects. In particular the big journals like science and nature seem to rely on a very small set of reviewers and often papers get reviewed by nonexperts, we had a paper rejected when a reviewer contradicted scientific fact that is in textbooks. When we protested the decision they just send it back to the same reviewer who stayed with his claim. I think if these reviews would be public the journals would get much clearer feedback on how bad some of the reviewers are.
- probably_wrong 6y ago> I think a first step is to make reviews public, in particular also rejects. As a reviewer, I have no objections towards my reviews going public. However, I have often had to tell authors that their paper is dumb (not in those terms, obviously), and I'd rather keep those reviews private. Being rejected is hard enough without the entire world knowing how much your paper sucks. Of course, there are terrible reviewers too. But IMHO that's better solved via better area chairs than with a Twitter mob. I would rather not being 4chan into the review process.
- DoreenMichele 6y agoFrom the article: Peer review grew up in a world where there were many fewer people engaging in science than today. Typically an editor would know everyone in the world who had contributed to the field in the past, and would have enough time to understand the ideas of each new entrant to the field as they started to submit papers. It relied on personal connections and deep and thoughtful understanding. That has changed just due to the scale of the scientific endeavor today, and is no longer possible in that form. I don't know how you fix that. But it points to why peer review worked well at one time and became some kind of gold standard and why it's failing to be some kind of gold standard today. I think you have to parse what was going right about the system that created peer review in order to have any hope of parsing how to replace it at scale for a world that's grown vastly larger in terms of sheer population numbers.
- barry-cotter 6y agoYes, peer review should be an informal process that happens after publication, not a formal one that turns a 40 page paper into a 100 page one and takes six years. Many fields have developed work arounds to peer review in the form of working papers, preprints and conferences. That this is necessary suggests that peer review be burned to the ground and the ashes sown with salt. https://www.stat.cmu.edu/~larry/Peer-Review.pdf https://www.stat.cmu.edu/~larry/Peer-Review.pdf > The refereeing process is very noisy, time consuming and arbitrary. We should be dissem- inating our research as widely as possible. Instead, we let two or three referees stand in between our work and the rest of our field. I think that most people are so used to our system, that they reflexively defend it when it is criticized. The purpose of doing research is to create new knowledge. This knowledge is useless unless it is disseminated. Refereeing is an impediment to dissemination. ... > Some will argue that refereeing provides quality control. This is an illusion. Plenty of bad papers get published and plenty of good papers get rejected. Many think that the stamp of approval by having a paper accepted by the refereeing process is crucial for maintaining the integrity of the field. This attitude treats a field as if it is a priesthood with a set of infallible, wise elders deciding what is good and what is bad. It is also like a guild, which protects itself by making it harder for outsiders to compete with insiders.
- etrautmann 6y agoYou’re creating a false dichotomy. It’s all about maximizing signal to noise, and expert reviewers are absolutely required (at some point in the process) regardless of whether they’re perfect.
- barry-cotter 6y agoWhat false dichotomy? If you want to keep journals revert to editorial review. If the editor thinks the paper isn’t garbage they publish it. Sometimes they’re wrong but you don’t get papers published six years after they’re written. Maximizing signal to noise is a terrible goal to put so much weight on. We should be trying to maximize the growth of knowledge, not minimizing the number of wrong papers, which is what max(s/n) leads to. Follow the editorial philosophy of Max Planck editing Annalen der Physik during the period when that was the pre-eminent journal in Physics during the most productive epoch for Physics research ever. > To shun much more the reproach of having suppressed strange opinions than that of having been too gentle in evaluating them.
- esfandia 6y agoA couple of ideas I've been mulling over for a while: - the quicker but less rigorous way: set things up like a forum like HackerNews, let the crowd-sourced upvote and comment system work its magic; make the karma/reputation points act as a proxy for the quality and relevance of the article, its authors, as well as the commenters. - the slower way: let articles "bubble" up from smaller but more specialized workshops toward more generalized and prestigious conferences and journals. Have the technical committee recommend the best articles to those more reputable venues, along with whatever it is the authors need to work on to re-submit the extended version at that "better" venue. This is kind of already the case with workshops or conferences that have an associated Special Issue of a journal; I'm advocating for a generalization of the practice to all venues and more than a 2-level hierarchy. The reputation of the lower venues could be backpropagated by the number of papers they successfully "bubble" up, PageRank style, so that would be the incentive the committees would have in recommending their papers to the other venues.
- randcraw 6y agoYour option 2 might work if the bubbling process is somehow made informed and fair, but option 1 is an invitation to promote charismatic loudmouths. Using pure populism as the means for seeking truth clearly doesn’t work very well, as the rising tide of of beloved trolls in media and politics attests.
- esfandia 6y agoFair point. There's a sweet spot to be found between full-on crowd-sourcing and giving all the power to 2-3 blind reviewers. Maybe limit the participants to be the members of the technical committee? Or maybe expand it to include members of the given conference?
- Vinnl 6y agoCurrently, peer review fulfils three distinct functions, to the detriment of all of them: 1. Acting as a filter for what research is worth reading. 2. Providing input to the author on how to improve their work. 3. Influence the author's professional evaluation. The first is the most blunt instrument: when something is not considered worth reading, it doesn't get published at all. This is problematic in the cases where the peer reviewers misjudge, or when an article is evaluated on more than its academic merit, e.g. whether it's "groundbreaking" enough for a particular journal. The second is useful, but made less so due to the power imbalance - an author cannot judge for themselves whether the feedback makes sense in the context of their work. This is especially problematic when someone has to take their work to multiple journals and/or gets conflicting feedback. And that power imbalance is the result of the third point: it can be more important to the author to get that stamp of approval by the reviewers than to meaningfully contribute to science, because that's the only way they can stay in academia. Ideally we'd split them up, but that does require aligning the academic community - which is a hard problem, especially considering there are strong vested interests in the status quo. (Disclosure: I do volunteer for a project (https://plaudit.pub https://plaudit.pub) aimed at splitting evaluation from publication, because I think it's an important issue.)
- j88439h84 6y agoInteresting list!
- albertzeyer 6y agoPlaudit is nothing more than like an upvote, right? I don't think this is really helpful. We already have a very similar metric, which is citation count. Which is definitely a useful metric. At first, I thought that Plaudit adds some post peer review platform, or discussion forum, for each paper. Similar to OpenReview but for any paper. I think this might actually be a great idea. I think that https://fermatslibrary.com/librarian https://fermatslibrary.com/librarian actually provides a similar functionality. But it has not seen much adoption.
- Vinnl 6y agoIt's not a metric, it's an endorsement. Think of it like being published in a journal: it's not about in how many journals an article gets published, but in which ones. Likewise, what's important here is who has endorsed it. Instead of a journal name acting as a proxy for "someone I trust thinks this work is worth reading", you can directly see the name of that person. (That said, a big challenge with citation counts is that they take a long time to accumulate.) As I mentioned, I think the three functions should be separated. Although Plaudit by itself does not facilitate giving feedback, it does not prevent it either. Ideally, if someone comes across e.g. an error in an article, they provide that feedback to the author (e.g. through an email, of using Hypothes.is - a project that I think you might like). There's no need to only contribute to improving scientific literature through a formal journal-assisted peer review process. And who knows: if the author incorporates that feedback, the giver of the feedback might decide to endorse that article using Plaudit :)
- timkam 6y agoI think we can learn from the OS community, but this requires changing the incentive system of academia. Two ideas: * Research papers should be living documents that are properly versioned; updates should get reviewed again. This obviously means that incremental improvements to papers and reviewing need to result in something tangible for one's CV. The versioning makes sure one can still point to a specific revision of a paper. Of course, updates should consider the implications of changes carefully (as is the case for software). For example, I might get familiar with a paper over the course of a month, and then use it as the cornerstone of one of my own works. Then, I want that the paper is stable and does not constantly mess with important formal definitions et cetera. Otherwise, people won't be able to agree on the implications the paper has for the community. * It might be better to make review and reviewers fully visible and move to a more interactive review process. For obvious reasons, practically "double-blind" review does not work, because if you're a visible member of your community and submit a paper, most people can guess that it's yours. If we would move from "one-shot" (or: one-shot plus rebuttal) reviews to more interactive, public, non-anonymous discussions, we would get rid of at least some of the social back-channeling that is only available to the bigshots in the community. Of course, I don't think we will get there quickly.
- sytelus 6y agoOne issue is that reviewers themselves don’t get rated. They also don’t have any incentives to do quality reviews. I think StackOverflow model is the only one I know of which is able to scale this process. It’s not without flaws but it’s content usability and quality is extremely high than any other collaborative filtering model out there. Academic reviews can perhaps be scaled with similar model. Imagine each paper is question and each review is answer. People vote for which answers and questions were useful and everyone accumulates reputation that in turn can be used for career development. You can see everyone’s questions, answers and votes. People with enough karma gets moderator privileges to be able to knock off bad content. People will then have incentive to do the right thing, improve quality and avoid favoritism.