13 ms·

>Reddit-like voting and commenting site for scientists would work just as well to vet papers Probably not. There's a well-known essay that explains why karma/

by jasode 9y ago

>Reddit-like voting and commenting site for scientists would work just as well to vet papers Probably not. There's a well-known essay that explains why karma/votes/likes work very well for easy tidbits like cat photos, gifs, and puns. (The stuff that takes less than 5 seconds to evaluate and requires near-zero effort to judge whether one "likes" it or not.) Unfortunately, I don't have a cite because I forgot who the author was. Longer topics such as a 20000 word dense research paper with supporting data that requires at least 2 hours of study by a reader/reviewer doesn't work for a karma points system. The effort is too high to contribute significant # of votes. It's the same difficulty for other long form content such as manuscripts for novels/books. Instead of karma voting, these works require curation that passes the "signal of approval" from one person to the next. With book manuscripts, curation is first done by the literary agents and then the acquisition editors. With scholarly papers, it's a somewhat analogous process (committee 1st pass then forwards to the peer reviewers). Each "approver" has a reputation which signals to the next person that it's ok to invest the hours into studying the work. This social signaling mechanism is also similar for startup entrepreneurs' business plans and VCs. VCs would rather be exposed to your business idea from a referral rather than an unsolicited cold email. All 3 domain examples of book manuscripts, academic research, and business plans share the same theme: people have limited time to wade through a vast sea of poor quality content. I think people can conclude that flipping through 100 cat photos is different than slogging through 100 dense papers. Another example of this dynamic is the Stackoverflow voting system. There have been several meta discussions lamenting the fact that the "easy" questions attracts a high volume of activity (answers + votes) but the "hard" questions languishes with little attention. That's because the easy questions are more analogous to the low-effort cat photos. Another commenter in the thread mentioned that github solved workflow for programmers and the same could be done for peer reviewing academic papers. This is misunderstanding the underlying issue. Evaluating dense papers is not an "information technology workflow problem" that's waiting for collaboration software to solve. The bottleneck is human effort to properly read papers to judge its correctness and significance. In 1687, a human would read Newton's Principia Mathematica at ~200 words-per-minute and after several hours, maybe he'd conclude that "yes, F=ma looks reasonable here". It's 300 years later and we're still the same slow humans reading papers at ~200 wpm. It's worse now because there's a larger glut of bad papers to wade through than before. Peer review of novels, scientific papers, business plans, is not solvable by a Github/Sharepoint/DropBox. The bottleneck isn't the copying of pdf or LaTeX files from one person to the next. It's the hours of human effort to judge the papers and signal to the next person to also expend hours of human effort. The issue about publishers isn't the filtering. The real issue is that they they charge too much money for filtering. The high profits is misleading some people to think it's an easy problem solved by github or a reddit-like voting system.

naasking 9y ago
> I think people can conclude that flipping through 100 cat photos is different than slogging through 100 dense papers. You're attacking a straw man. I said reddit-like, not reddit-clone. There are plenty of obvious problems with reddit's voting system. So what? What is scientific consensus if not a vote among some random sampling of scientists studying a field. What's left is to formalize this process in an open system so we're not beholden to the publishers and other parties with perverse incentives. > Evaluating dense papers is not an "information technology workflow problem" that's waiting for collaboration software to solve. The bottleneck is human effort to properly read papers to judge its correctness and significance. You're wrong. Firstly, it's often impossible to judge the significance of a result even soon after publishing. Secondly, reviewers already donate their time to review papers, so the human effort is already there. What's needed is a well-recognized system encoding a publishing process, whereby some random selection of scientists obtain access to early drafts with a way to provide commentary and ratings on various metrics, with final publication after some quality consensus is reached among the random selection of reviewers (as a rough idea). This is exactly an information technology problem.
- jasode 9y ago
  > I said reddit-like, not reddit-clone. Yes, and that's what I understood you to mean and a reddit-like voting system for papers is what I was talking about. (I even wrote "reddit-like voting system" as my last sentence in the post you replied to.) >You're wrong. Firstly, it's often impossible to judge the significance of a result even soon after publishing. I'm not talking about "significance" in relation to the whole of human history. I mean "significance" in the sense of a threshold for requesting other people to invest their time & effort to study the paper and also the subsequent subscribers to read. >, reviewers already donate their time to review papers, so the human effort is already there. Yes, but the key you're missing is that they donate their time because a respectable _somebody_ requested them to do so. The triggers for motivating that donated effort matters. If I'm a working scientist working 50 hours a week, I'm not logging into a reddit-like voting system and investing hours to read (mostly bad) papers that have 0 votes so I can submit my 1 vote. No, somebody I respect (e.g. a journal editor) needs to ping me and say, "hey, this paper looks interesting, can you review it?" After I peer review it, I may get some quid-pro-quo such as free access to the journal's other papers and/or when it's my turn to submit a paper, the editor will consider forwarding it. So yes, I'm not "paid in cash" but I'm paid "in kind" with some intangibles. That social dynamic is not replicable in a reddit-like voting system. Therefore your following scenario of: >whereby some random selection of scientists obtain access to early drafts ... is not realistic because scientists do not want to be part of random pool to be assigned random papers. That's contrary to basic human nature! Scientists want papers referred to them by another human and not an algorithm. It's not a technology problem, it's a human problem. All the software suggestions to solve this from programmers including ideas such as a reddit-like system, or a github-like system, or a blockchain type system, etc ... they all misunderstand the culture and psychological motivations of publishing. >What is scientific consensus if not a vote among some random sampling of scientists studying a field. Yes, I can see that if you reduce the problem to the simple abstraction of "votes", it seems like building a software platform to formally capture votes is The Solution. What your simplification leaves out is all the human signaling required before people render their votes. That's the human dynamic that collaboration software doesn't solve. (In any case, an accepted published paper isn't "consensus" per se; it's just another piece of evidence for people to weigh. The paper just passed through enough human curation filters that may be worthwhile to read.)

by jasode 9y ago

naasking 9y ago

> I think people can conclude that flipping through 100 cat photos is different than slogging through 100 dense papers. You're attacking a straw man. I said reddit-like, not reddit-clone. There are plenty of obvious problems with reddit's voting system. So what? What is scientific consensus if not a vote among some random sampling of scientists studying a field. What's left is to formalize this process in an open system so we're not beholden to the publishers and other parties with perverse incentives. > Evaluating dense papers is not an "information technology workflow problem" that's waiting for collaboration software to solve. The bottleneck is human effort to properly read papers to judge its correctness and significance. You're wrong. Firstly, it's often impossible to judge the significance of a result even soon after publishing. Secondly, reviewers already donate their time to review papers, so the human effort is already there. What's needed is a well-recognized system encoding a publishing process, whereby some random selection of scientists obtain access to early drafts with a way to provide commentary and ratings on various metrics, with final publication after some quality consensus is reached among the random selection of reviewers (as a rough idea). This is exactly an information technology problem.

jasode 9y ago
> I said reddit-like, not reddit-clone. Yes, and that's what I understood you to mean and a reddit-like voting system for papers is what I was talking about. (I even wrote "reddit-like voting system" as my last sentence in the post you replied to.) >You're wrong. Firstly, it's often impossible to judge the significance of a result even soon after publishing. I'm not talking about "significance" in relation to the whole of human history. I mean "significance" in the sense of a threshold for requesting other people to invest their time & effort to study the paper and also the subsequent subscribers to read. >, reviewers already donate their time to review papers, so the human effort is already there. Yes, but the key you're missing is that they donate their time because a respectable _somebody_ requested them to do so. The triggers for motivating that donated effort matters. If I'm a working scientist working 50 hours a week, I'm not logging into a reddit-like voting system and investing hours to read (mostly bad) papers that have 0 votes so I can submit my 1 vote. No, somebody I respect (e.g. a journal editor) needs to ping me and say, "hey, this paper looks interesting, can you review it?" After I peer review it, I may get some quid-pro-quo such as free access to the journal's other papers and/or when it's my turn to submit a paper, the editor will consider forwarding it. So yes, I'm not "paid in cash" but I'm paid "in kind" with some intangibles. That social dynamic is not replicable in a reddit-like voting system. Therefore your following scenario of: >whereby some random selection of scientists obtain access to early drafts ... is not realistic because scientists do not want to be part of random pool to be assigned random papers. That's contrary to basic human nature! Scientists want papers referred to them by another human and not an algorithm. It's not a technology problem, it's a human problem. All the software suggestions to solve this from programmers including ideas such as a reddit-like system, or a github-like system, or a blockchain type system, etc ... they all misunderstand the culture and psychological motivations of publishing. >What is scientific consensus if not a vote among some random sampling of scientists studying a field. Yes, I can see that if you reduce the problem to the simple abstraction of "votes", it seems like building a software platform to formally capture votes is The Solution. What your simplification leaves out is all the human signaling required before people render their votes. That's the human dynamic that collaboration software doesn't solve. (In any case, an accepted published paper isn't "consensus" per se; it's just another piece of evidence for people to weigh. The paper just passed through enough human curation filters that may be worthwhile to read.)