4 ms·
Figuring out what to replace the current system with is a non-trivial problem. Obviously replacing it with a simple popularity contest is not the right answer.
by lisper 5y ago
Figuring out what to replace the current system with is a non-trivial problem. Obviously replacing it with a simple popularity contest is not the right answer. But figuring out what the right answer is is exactly the sort of thing that the academy ought to be good at, or at least attempting, but they aren't even trying.
I think the Right Answer would look something like Page Rank, except with reviewers being ranked. Anyone can review, but the relative weight of each review is biased according to that reviewer's reputation, which in turn is based on the collective value of the reviews they have received. There would still be a role for some kind of governing body, but its function would be to enforce continuity of identity and the binding of identity to publications. So something like Arxiv, but with a vetting process to insure that a paper or review submitted by an author really was written by that author, and to prevent one person from publishing with multiple identities.
- thfuran 5y agoHow would you weight say a highly ranked computational biologist reviewing a paper in cultural anthropology?
- j-pb 5y agoI'd argue that such a system would be much better at avoiding the current political swamp, if the reputation/trust was relative to your personal trust into certain individuals and research directions. E.g. we might view different close people as trustworthy or judge their research with different value, which in turn have their subjective judgement. That subjective judgement is inherent in the scientific method and allows different opinions to compete until the truth emerges. I think that the web of trust should reflect that and give you a personalized score for each paper based on a trust network between public keys.
- dekhn 5y agoI'm not opposed to what you're saying, but be aware that Page Rank literally came from scientific citation indices which computed similar centrality measures long before page rank (I don't get why everybody thinks page rank was particularly clever or interesting; centrality measures were an active area of research well before that, and many people were epxloring applying it to web ranking at the time). And we have some modestly successful sites that kind-of do what you're saying (Faculty of 1000: specifically F1000 Research).
- lupire 5y agoPage rank was interesting because it's a little hard to do at scale, and because no other search engine would do it (because it's not profitable -- AdWords is main innovation behind making PageRank practical)
- dekhn 5y agopage rank isn't hard to do at scale- it was an iterative map reduce. It also predates adwords by several year (PR paper published in 98, already live by that time, AdWords launched in 2000). Ask.com used HITS which was published in 99.
- scarmig 5y ago> Anyone can review, but the relative weight of each review is biased according to that reviewer's reputation, which in turn is based on the collective value of the reviews they have received. Someone could be very good at reviewing papers but not particularly skilled at original research, or vice versa. They're different skill sets. Also, giving the most cited authors the most power would have a homogenizing effect: the best way to get ahead would then be to push things that support the viewpoints of those currently at the top of the field. (I agree that something needs to change; I'm better at critiquing ideas than generating my own...)
- borroka 5y agoIt may not be trivial to replace the current system. But the current system is certainly not a local maximum. As a former academic who has been doing research for more than 15 years, it's hard to reconcile the idea of a local maximum, or, say, a system for advancing knowledge that works well, with (a) submitting an article to a journal, (b) sending it in for review, (c) getting a rejection notice with reviews that don't make sense to the point where you get the sense that they've reviewed another article, (d) sending an appeal to the Associate Editor making it clear that the reviews don't make sense, (e) having the AE respond that they themselves have read the article and agree with the reviewers. Most people, especially outside of academia, think that the problems with the peer review process are something more "philosophical" than they are, like the old establishment not wanting the new generation of physicists to present and support a new "Standard Model". But the real problems are more mundane, and consist of thousands of people wasting time and energy following processes that are, for the most part, crazy.
- lisper 5y agoI think it's even more fundamental than that. The current system is binary. An article either is published or it is not. There are nuances beyond this, like citation counts and journal rankings, but "published in a peer-reviewed journal" is a mantra that is often taken as a synonym for "established as truth." Whatever replaces the current system ought to have an output that is more like, "Lots of people who have published things that lots of people think are worthy of note think that this paper is worthy of note."
- quantum_magpie 5y agoI'm a big fan of Atmospheric Chemistry and Physics model, where they archive and make public everything throughout the publication process: preprint, peer reviews, and author responses. https://www.atmospheric-chemistry-and-physics.net/ https://www.atmospheric-chemistry-and-physics.net/