6 ms·

> The whole idea of a journal should be obsolete. > A tenured professor is still a highly respected position despite the fact that the actual job of being a pr

> The whole idea of a journal should be obsolete. > A tenured professor is still a highly respected position despite the fact that the actual job of being a professor has a lot more to do with politics than it does with academics It seems to me that given your 2 above statements you essentially want to abolish journals and peer review in order to be able to give researchers the ability to sponsor, parlay and generally use their social skills to push a paper out. You mention the internet so it's granted that as soon as somebody discovers that the more rabid followers one has, the better it is for the papers one is pushing...it becomes an arms race All of the above is in order to essentially enable researchers to do what you accuse professors of doing (using politics to get ahead without or with little contribution).

lisper 4y ago
Figuring out what to replace the current system with is a non-trivial problem. Obviously replacing it with a simple popularity contest is not the right answer. But figuring out what the right answer is is exactly the sort of thing that the academy ought to be good at, or at least attempting, but they aren't even trying. I think the Right Answer would look something like Page Rank, except with reviewers being ranked. Anyone can review, but the relative weight of each review is biased according to that reviewer's reputation, which in turn is based on the collective value of the reviews they have received. There would still be a role for some kind of governing body, but its function would be to enforce continuity of identity and the binding of identity to publications. So something like Arxiv, but with a vetting process to insure that a paper or review submitted by an author really was written by that author, and to prevent one person from publishing with multiple identities.
- thfuran 4y ago
  How would you weight say a highly ranked computational biologist reviewing a paper in cultural anthropology?
  - j-pb 4y ago
    Meta reviews.
  - lisper 4y ago
    That is a really excellent question. That's why I said "something like Page Rank" and not just Page Rank. This is an unsolved problem, but I don't think it's an unsolvable problem.
- j-pb 4y ago
  I'd argue that such a system would be much better at avoiding the current political swamp, if the reputation/trust was relative to your personal trust into certain individuals and research directions. E.g. we might view different close people as trustworthy or judge their research with different value, which in turn have their subjective judgement. That subjective judgement is inherent in the scientific method and allows different opinions to compete until the truth emerges. I think that the web of trust should reflect that and give you a personalized score for each paper based on a trust network between public keys.
- dekhn 4y ago
  I'm not opposed to what you're saying, but be aware that Page Rank literally came from scientific citation indices which computed similar centrality measures long before page rank (I don't get why everybody thinks page rank was particularly clever or interesting; centrality measures were an active area of research well before that, and many people were epxloring applying it to web ranking at the time). And we have some modestly successful sites that kind-of do what you're saying (Faculty of 1000: specifically F1000 Research).
  - lupire 4y ago
    Page rank was interesting because it's a little hard to do at scale, and because no other search engine would do it (because it's not profitable -- AdWords is main innovation behind making PageRank practical)
    - dekhn 4y ago
      page rank isn't hard to do at scale- it was an iterative map reduce. It also predates adwords by several year (PR paper published in 98, already live by that time, AdWords launched in 2000). Ask.com used HITS which was published in 99.
- scarmig 4y ago
  > Anyone can review, but the relative weight of each review is biased according to that reviewer's reputation, which in turn is based on the collective value of the reviews they have received. Someone could be very good at reviewing papers but not particularly skilled at original research, or vice versa. They're different skill sets. Also, giving the most cited authors the most power would have a homogenizing effect: the best way to get ahead would then be to push things that support the viewpoints of those currently at the top of the field. (I agree that something needs to change; I'm better at critiquing ideas than generating my own...)
- borroka 4y ago
  It may not be trivial to replace the current system. But the current system is certainly not a local maximum. As a former academic who has been doing research for more than 15 years, it's hard to reconcile the idea of a local maximum, or, say, a system for advancing knowledge that works well, with (a) submitting an article to a journal, (b) sending it in for review, (c) getting a rejection notice with reviews that don't make sense to the point where you get the sense that they've reviewed another article, (d) sending an appeal to the Associate Editor making it clear that the reviews don't make sense, (e) having the AE respond that they themselves have read the article and agree with the reviewers. Most people, especially outside of academia, think that the problems with the peer review process are something more "philosophical" than they are, like the old establishment not wanting the new generation of physicists to present and support a new "Standard Model". But the real problems are more mundane, and consist of thousands of people wasting time and energy following processes that are, for the most part, crazy.
  - lisper 4y ago
    I think it's even more fundamental than that. The current system is binary. An article either is published or it is not. There are nuances beyond this, like citation counts and journal rankings, but "published in a peer-reviewed journal" is a mantra that is often taken as a synonym for "established as truth." Whatever replaces the current system ought to have an output that is more like, "Lots of people who have published things that lots of people think are worthy of note think that this paper is worthy of note."
- quantum_magpie 4y ago
  I'm a big fan of Atmospheric Chemistry and Physics model, where they archive and make public everything throughout the publication process: preprint, peer reviews, and author responses. https://www.atmospheric-chemistry-and-physics.net/ https://www.atmospheric-chemistry-and-physics.net/
j-pb 4y ago
Your argument doesn't hold economically nor based on the state of the status quo. Economically it is infeasible to politically play, social engineer, or bribe thousands of reviewers in a publicly inspectable web of trust. It's easily done so and done in a secretive environment with a handfull of people that all know each other, i.e. the current state of publishing. The current paper review is a farce, you just go conference and review shopping till it gets accepted, papers get shitgunned into half a dozen journals with minor changes, and reviewers often have no clue about contents and just check the general form, wave papers from friendlies through, or block competitors based on personal vendettas and funding rivalries. Publishing and review in a decentralized anonymised web of trust secured by crypto is probably the only valid use case for crypto and blockchain foo that I know of.
- I_DRINK_KOOLAID 4y ago
  > Economically it is infeasible to politically play, social engineer, or bribe thousands of reviewers in a publicly inspectable web of trust Elon Musk hyperloop whitepaper pipedream...he made it in one night, probably on heavy drugs. It got press for 8 years, still getting press. A bastardised version of Hyperloop got hundreds of millions in public funding in Las Vegas and Los Angeles for what essentially amounts to a large underground car garage with skeets. The company selling this bastardised version just raised 700M in equity. "infeasible". When you open the floodgates of the court of public opinion...it becomes the game of the person who screams the loudest and is able to con people with the most confidence or has the loudest megaphone. History has shown us time and time again that the best governance system is a bunch of elder statesmen who are terrified of being judged harshly by their peers long after they are gone. For example people still say about Newton that his track record wasn't immaculate because he wasted time looking for hidden messages in the Bible or trying to turn bronze into gold. Same for Einstein, his track record isn't immaculate because he was wrong on quantum-mechanics. Elder statesmen who dream of having the track record of a Pythagoras, that's the perfect governance system, that just requires finding them, doesn't require crypto-tokens or blockchain or any other flavor du jour tech.
  - j-pb 4y ago
glitchc 4y ago
The current system is the one where researchers sponsor, parlay and generally use their social skills to push a paper out. Appeal to authority is a strong signal determining paper acceptance as well. Cliques are common-place, and folks self-select venues based on their connections. The current publication system is indeed broken as it stands.

lisper 4y ago

Figuring out what to replace the current system with is a non-trivial problem. Obviously replacing it with a simple popularity contest is not the right answer. But figuring out what the right answer is is exactly the sort of thing that the academy ought to be good at, or at least attempting, but they aren't even trying. I think the Right Answer would look something like Page Rank, except with reviewers being ranked. Anyone can review, but the relative weight of each review is biased according to that reviewer's reputation, which in turn is based on the collective value of the reviews they have received. There would still be a role for some kind of governing body, but its function would be to enforce continuity of identity and the binding of identity to publications. So something like Arxiv, but with a vetting process to insure that a paper or review submitted by an author really was written by that author, and to prevent one person from publishing with multiple identities.

thfuran 4y ago
How would you weight say a highly ranked computational biologist reviewing a paper in cultural anthropology?
- j-pb 4y ago
  Meta reviews.
- lisper 4y ago
  That is a really excellent question. That's why I said "something like Page Rank" and not just Page Rank. This is an unsolved problem, but I don't think it's an unsolvable problem.
j-pb 4y ago
I'd argue that such a system would be much better at avoiding the current political swamp, if the reputation/trust was relative to your personal trust into certain individuals and research directions. E.g. we might view different close people as trustworthy or judge their research with different value, which in turn have their subjective judgement. That subjective judgement is inherent in the scientific method and allows different opinions to compete until the truth emerges. I think that the web of trust should reflect that and give you a personalized score for each paper based on a trust network between public keys.
dekhn 4y ago
I'm not opposed to what you're saying, but be aware that Page Rank literally came from scientific citation indices which computed similar centrality measures long before page rank (I don't get why everybody thinks page rank was particularly clever or interesting; centrality measures were an active area of research well before that, and many people were epxloring applying it to web ranking at the time). And we have some modestly successful sites that kind-of do what you're saying (Faculty of 1000: specifically F1000 Research).
- lupire 4y ago
  Page rank was interesting because it's a little hard to do at scale, and because no other search engine would do it (because it's not profitable -- AdWords is main innovation behind making PageRank practical)
  - dekhn 4y ago
    page rank isn't hard to do at scale- it was an iterative map reduce. It also predates adwords by several year (PR paper published in 98, already live by that time, AdWords launched in 2000). Ask.com used HITS which was published in 99.
scarmig 4y ago
> Anyone can review, but the relative weight of each review is biased according to that reviewer's reputation, which in turn is based on the collective value of the reviews they have received. Someone could be very good at reviewing papers but not particularly skilled at original research, or vice versa. They're different skill sets. Also, giving the most cited authors the most power would have a homogenizing effect: the best way to get ahead would then be to push things that support the viewpoints of those currently at the top of the field. (I agree that something needs to change; I'm better at critiquing ideas than generating my own...)
borroka 4y ago
It may not be trivial to replace the current system. But the current system is certainly not a local maximum. As a former academic who has been doing research for more than 15 years, it's hard to reconcile the idea of a local maximum, or, say, a system for advancing knowledge that works well, with (a) submitting an article to a journal, (b) sending it in for review, (c) getting a rejection notice with reviews that don't make sense to the point where you get the sense that they've reviewed another article, (d) sending an appeal to the Associate Editor making it clear that the reviews don't make sense, (e) having the AE respond that they themselves have read the article and agree with the reviewers. Most people, especially outside of academia, think that the problems with the peer review process are something more "philosophical" than they are, like the old establishment not wanting the new generation of physicists to present and support a new "Standard Model". But the real problems are more mundane, and consist of thousands of people wasting time and energy following processes that are, for the most part, crazy.
- lisper 4y ago
  I think it's even more fundamental than that. The current system is binary. An article either is published or it is not. There are nuances beyond this, like citation counts and journal rankings, but "published in a peer-reviewed journal" is a mantra that is often taken as a synonym for "established as truth." Whatever replaces the current system ought to have an output that is more like, "Lots of people who have published things that lots of people think are worthy of note think that this paper is worthy of note."
quantum_magpie 4y ago
I'm a big fan of Atmospheric Chemistry and Physics model, where they archive and make public everything throughout the publication process: preprint, peer reviews, and author responses. https://www.atmospheric-chemistry-and-physics.net/ https://www.atmospheric-chemistry-and-physics.net/

j-pb 4y ago

Your argument doesn't hold economically nor based on the state of the status quo. Economically it is infeasible to politically play, social engineer, or bribe thousands of reviewers in a publicly inspectable web of trust. It's easily done so and done in a secretive environment with a handfull of people that all know each other, i.e. the current state of publishing. The current paper review is a farce, you just go conference and review shopping till it gets accepted, papers get shitgunned into half a dozen journals with minor changes, and reviewers often have no clue about contents and just check the general form, wave papers from friendlies through, or block competitors based on personal vendettas and funding rivalries. Publishing and review in a decentralized anonymised web of trust secured by crypto is probably the only valid use case for crypto and blockchain foo that I know of.

I_DRINK_KOOLAID 4y ago
> Economically it is infeasible to politically play, social engineer, or bribe thousands of reviewers in a publicly inspectable web of trust Elon Musk hyperloop whitepaper pipedream...he made it in one night, probably on heavy drugs. It got press for 8 years, still getting press. A bastardised version of Hyperloop got hundreds of millions in public funding in Las Vegas and Los Angeles for what essentially amounts to a large underground car garage with skeets. The company selling this bastardised version just raised 700M in equity. "infeasible". When you open the floodgates of the court of public opinion...it becomes the game of the person who screams the loudest and is able to con people with the most confidence or has the loudest megaphone. History has shown us time and time again that the best governance system is a bunch of elder statesmen who are terrified of being judged harshly by their peers long after they are gone. For example people still say about Newton that his track record wasn't immaculate because he wasted time looking for hidden messages in the Bible or trying to turn bronze into gold. Same for Einstein, his track record isn't immaculate because he was wrong on quantum-mechanics. Elder statesmen who dream of having the track record of a Pythagoras, that's the perfect governance system, that just requires finding them, doesn't require crypto-tokens or blockchain or any other flavor du jour tech.
- j-pb 4y ago

glitchc 4y ago

The current system is the one where researchers sponsor, parlay and generally use their social skills to push a paper out. Appeal to authority is a strong signal determining paper acceptance as well. Cliques are common-place, and folks self-select venues based on their connections. The current publication system is indeed broken as it stands.