7 ms·

I'm missing the leap here: > The taxpayer ends up paying twice for the same research: once to fund it and a second time to read it. The heart of the problem li

by mjn 14y ago

I'm missing the leap here: > The taxpayer ends up paying twice for the same research: once to fund it and a second time to read it. The heart of the problem lies in the reputation system, which encourages scientists to put their work behind paywalls. The way out of this mess is to build new reputation metrics. The heart of the problem simply lies in closed-access journals, and a simpler solution is to move to open-access journals, which is already happening. Reputation is not an insurmountable barrier to this. In my field (artificial intelligence) the top two journals are now open-access. No fancy new metrics or removal of the concept of journal needed; problem solved. I view the solutions proposed here as a bit more problematic. Download counts in particular are likely to just reward linkbaity research, rather than quality research, and exacerbate the already problematic race to put out misleading hyped-up press releases. Citation counts are liable to gaming as well, and are commonly gamed by both citation rings, and by people who consciously choose the subjects they publish in an ADDish citation-maximizing way, jumping in and out of hot areas to drop off a paper that'll turn up in searches. When it comes to judging the quality of AI research, I have a lot more trust in the editorial process of the open-access journals like JMLR or JAIR than I do in gimmicky new metrics that try to reconceptualize scientific publishing as gamification, with all the baggage gamification brings (mostly notably that it's not openly open to, but positively encourages treating it as a points-grabbing system to be gamed). I don't have much trust in the supposedly "open" motives of this new batch of para-academic for-profit companies, either. Notice how academia.edu won't even let you download the PDFs of articles without registering for an account. I would guess the real purpose of these metrics is to set themselves up as new scientific gatekeepers in one form or another: to transition from a journal-oriented publication system to an "academic marketplace" oriented system where they own the marketplace. I have a lot more trust in the by-scientists, for-scientists model of JMLR.

RichardPrice 14y ago
Open access journals, based on authors paying $1,000-$3,000 to publish a paper, are going to be a temporary stepping stone on the way to the decline of the journal. We need new reputation metrics to encourage the sharing of data-sets, videos, code, and generally the full range of a scientist's output. 75% of the world's scientific data doesn't get shared because the rewards aren't there for the scientist to share it. Journals aren't going to start publishing data-sets (or code, or videos, or blog posts), and so we need new reputation metrics, and reward systems, to encourage scientists to share this kind of information. The other problem with the journal system is that the peer review system is slow and surfaces the opinions of only two or three scientists about a given paper. If you have a community of 20,000 people in the DNA sequencing research community, asking two people what they think about a paper is not going to deliver a statistically significant result. Furthermore the typical time-lag for the system to surface those opinions is 12 months. We need a more robust peer review system which surfaces opinions from the entire scientific community in real time. Reputation metrics are the key to building that system. We need to provide the rewards for scientists to share their comments on papers, so they can build their reputations off the insights they share on each other's papers.
- mjn 14y ago
  Yes, I get that's the sales pitch, but you didn't address gamification at all, which is what you're promoting, and will positively harm science, not improve it. In addition, peer-review at well-run open access journals, which are the model I think we should be moving towards, does not take anywhere near 12 months, and in CS there are no publication fees (I gather this differs in biology). I typically get very well-thought-out reviews which help to improve my papers in about 6-8 weeks, and if accepted, publication is completely free of charge. And, it's becoming increasingly common to circulate preprints ahead of submission, on places such as the arXiv or at conferences or workshops, to get the feedback of a larger proportion of the community before publishing the final version of the paper. I agree peer review does not settle the correctness of a paper; it's a system for ensuring a high-quality discourse, not to publish only papers which are the last word on a subject. That's what reply papers, replication studies, etc. are for. I am skeptical that attaching a bunch of points and badges to the discourse is going to improve the quality of that discourse, and suspect it will harm it. I'd rather look at how to improve the discourse by improving the journals. But that won't make money for VCs, so I'm not surprised that we're only seeing it from nonprofit organizations like JAIR and PLoS.
  - RichardPrice 14y ago
    With the time-lag, you have to take into account the rejection rate. Most articles don't get into their first choice journal, and perhaps get into their second or third choice. You can't parallel submit, meaning that the time-lag between finishing the paper, and it getting published in a journal, often ranges between 6 and 18 months. I understand that Computer Science does have a slightly faster peer review process than other fields. Here is a piece where a leading neuroscientist writes "Today the lag [in my field] between submission and acceptance is often more than a year". http://www.pnas.org/content/early/2013/01/25/201300924.short?rss=1 http://www.pnas.org/content/early/2013/01/25/201300924.short... Gamification is an important issue. You should note that the journal's peer review system, based on two people reading a paper, gets gamed. There is the wide-spread practice of defensive citation, where you defensively cite anyone who might conceivably peer review your paper, in order to maximize the chances of its getting accepted. Some people know how to play that game better than others. One way of handling gamification is to look at the quality of the people interacting with a piece of content, and not just the quantity. This is how Google's PageRank algorithm deals with the issue. There is also the approach, taken by Twitter and Facebook, of picking out the people's whose judgements you trust, and then letting them act as quality filters. Those systems can be gamed too, but aren't typically gamed to a large extent.

by mjn 14y ago

RichardPrice 14y ago

Open access journals, based on authors paying $1,000-$3,000 to publish a paper, are going to be a temporary stepping stone on the way to the decline of the journal. We need new reputation metrics to encourage the sharing of data-sets, videos, code, and generally the full range of a scientist's output. 75% of the world's scientific data doesn't get shared because the rewards aren't there for the scientist to share it. Journals aren't going to start publishing data-sets (or code, or videos, or blog posts), and so we need new reputation metrics, and reward systems, to encourage scientists to share this kind of information. The other problem with the journal system is that the peer review system is slow and surfaces the opinions of only two or three scientists about a given paper. If you have a community of 20,000 people in the DNA sequencing research community, asking two people what they think about a paper is not going to deliver a statistically significant result. Furthermore the typical time-lag for the system to surface those opinions is 12 months. We need a more robust peer review system which surfaces opinions from the entire scientific community in real time. Reputation metrics are the key to building that system. We need to provide the rewards for scientists to share their comments on papers, so they can build their reputations off the insights they share on each other's papers.

mjn 14y ago
Yes, I get that's the sales pitch, but you didn't address gamification at all, which is what you're promoting, and will positively harm science, not improve it. In addition, peer-review at well-run open access journals, which are the model I think we should be moving towards, does not take anywhere near 12 months, and in CS there are no publication fees (I gather this differs in biology). I typically get very well-thought-out reviews which help to improve my papers in about 6-8 weeks, and if accepted, publication is completely free of charge. And, it's becoming increasingly common to circulate preprints ahead of submission, on places such as the arXiv or at conferences or workshops, to get the feedback of a larger proportion of the community before publishing the final version of the paper. I agree peer review does not settle the correctness of a paper; it's a system for ensuring a high-quality discourse, not to publish only papers which are the last word on a subject. That's what reply papers, replication studies, etc. are for. I am skeptical that attaching a bunch of points and badges to the discourse is going to improve the quality of that discourse, and suspect it will harm it. I'd rather look at how to improve the discourse by improving the journals. But that won't make money for VCs, so I'm not surprised that we're only seeing it from nonprofit organizations like JAIR and PLoS.
- RichardPrice 14y ago
  With the time-lag, you have to take into account the rejection rate. Most articles don't get into their first choice journal, and perhaps get into their second or third choice. You can't parallel submit, meaning that the time-lag between finishing the paper, and it getting published in a journal, often ranges between 6 and 18 months. I understand that Computer Science does have a slightly faster peer review process than other fields. Here is a piece where a leading neuroscientist writes "Today the lag [in my field] between submission and acceptance is often more than a year". http://www.pnas.org/content/early/2013/01/25/201300924.short?rss=1 http://www.pnas.org/content/early/2013/01/25/201300924.short... Gamification is an important issue. You should note that the journal's peer review system, based on two people reading a paper, gets gamed. There is the wide-spread practice of defensive citation, where you defensively cite anyone who might conceivably peer review your paper, in order to maximize the chances of its getting accepted. Some people know how to play that game better than others. One way of handling gamification is to look at the quality of the people interacting with a piece of content, and not just the quantity. This is how Google's PageRank algorithm deals with the issue. There is also the approach, taken by Twitter and Facebook, of picking out the people's whose judgements you trust, and then letting them act as quality filters. Those systems can be gamed too, but aren't typically gamed to a large extent.