3 ms·

Open access journals, based on authors paying $1,000-$3,000 to publish a paper, are going to be a temporary stepping stone on the way to the decline of the jour

Open access journals, based on authors paying $1,000-$3,000 to publish a paper, are going to be a temporary stepping stone on the way to the decline of the journal. We need new reputation metrics to encourage the sharing of data-sets, videos, code, and generally the full range of a scientist's output. 75% of the world's scientific data doesn't get shared because the rewards aren't there for the scientist to share it. Journals aren't going to start publishing data-sets (or code, or videos, or blog posts), and so we need new reputation metrics, and reward systems, to encourage scientists to share this kind of information. The other problem with the journal system is that the peer review system is slow and surfaces the opinions of only two or three scientists about a given paper. If you have a community of 20,000 people in the DNA sequencing research community, asking two people what they think about a paper is not going to deliver a statistically significant result. Furthermore the typical time-lag for the system to surface those opinions is 12 months. We need a more robust peer review system which surfaces opinions from the entire scientific community in real time. Reputation metrics are the key to building that system. We need to provide the rewards for scientists to share their comments on papers, so they can build their reputations off the insights they share on each other's papers.

mjn 14y ago
Yes, I get that's the sales pitch, but you didn't address gamification at all, which is what you're promoting, and will positively harm science, not improve it. In addition, peer-review at well-run open access journals, which are the model I think we should be moving towards, does not take anywhere near 12 months, and in CS there are no publication fees (I gather this differs in biology). I typically get very well-thought-out reviews which help to improve my papers in about 6-8 weeks, and if accepted, publication is completely free of charge. And, it's becoming increasingly common to circulate preprints ahead of submission, on places such as the arXiv or at conferences or workshops, to get the feedback of a larger proportion of the community before publishing the final version of the paper. I agree peer review does not settle the correctness of a paper; it's a system for ensuring a high-quality discourse, not to publish only papers which are the last word on a subject. That's what reply papers, replication studies, etc. are for. I am skeptical that attaching a bunch of points and badges to the discourse is going to improve the quality of that discourse, and suspect it will harm it. I'd rather look at how to improve the discourse by improving the journals. But that won't make money for VCs, so I'm not surprised that we're only seeing it from nonprofit organizations like JAIR and PLoS.
- RichardPrice 14y ago
  With the time-lag, you have to take into account the rejection rate. Most articles don't get into their first choice journal, and perhaps get into their second or third choice. You can't parallel submit, meaning that the time-lag between finishing the paper, and it getting published in a journal, often ranges between 6 and 18 months. I understand that Computer Science does have a slightly faster peer review process than other fields. Here is a piece where a leading neuroscientist writes "Today the lag [in my field] between submission and acceptance is often more than a year". http://www.pnas.org/content/early/2013/01/25/201300924.short?rss=1 http://www.pnas.org/content/early/2013/01/25/201300924.short... Gamification is an important issue. You should note that the journal's peer review system, based on two people reading a paper, gets gamed. There is the wide-spread practice of defensive citation, where you defensively cite anyone who might conceivably peer review your paper, in order to maximize the chances of its getting accepted. Some people know how to play that game better than others. One way of handling gamification is to look at the quality of the people interacting with a piece of content, and not just the quantity. This is how Google's PageRank algorithm deals with the issue. There is also the approach, taken by Twitter and Facebook, of picking out the people's whose judgements you trust, and then letting them act as quality filters. Those systems can be gamed too, but aren't typically gamed to a large extent.
  - jacquesm 14y ago
    I think you are confusing gamification and gaming the system, those are not the same thing. GP was referring to badges, points etc (gamification) and you are referring to gaming the system (ie citing defensively).
    - jessriedel 14y ago
      If so, then mjn needs to clearly distinguish between (bad) gamification and a (good) reputation system, which will necessarily involve something like points or certifications.
  - notahacker 14y ago
    "Defensive" citations are arguably consistent with good research practice in that they reference all relevant research by the most prominent names in the field; it's certainly no worse than the sort of open trading of citations a PageRank style algorithm theoretically encourages. Sorry, I don't think Twitter and Facebook's quality filters "aren't typically gamed to a large extent" and I do think they're incomparably worse.
jessriedel 14y ago
> Open access journals, based on authors paying $1,000-$3,000 to publish a paper, are going to be a temporary stepping stone on the way to the decline of the journal. At least in my field, papers cost roughly $50k-$100k for just the researcher salary and overhead. Paying $2k to get a paper published is not a high hurdle (especially once the norm of closed-access publishing fades away). And this really seems like the minimum cost that needs to be incurred in order to get decent copyediting and the editorial cost of co-ordinating the referees. I love the ArXiv, but if I can get a the copyedited version from the journal I will. Yes the referees aren't paid, which may or may not change, but paying a PhD editor $500 to figure out the best referees and engage in the back-and-forth with them and the authors is pretty minimal. So $2k for a professionally edited, polished paper which is then free to everyone forever seems like a pretty good deal, and essentially a non-issue. Your point about the need for incentives for scientists to share non-paper scientific output like code is well-taken (and well-known). But I don't see why this can't be solved by open-access journals. I am completely unconvinced that facebook likes controlled by a for-profit organization are the way to go. EDIT: By the way, you've mentioned in several places the 12-month time lag for publishing. This was solved for the physics community by the ArXiv; the papers get immediately released, and you can still submit to journals for the benefits of copyediting and refereeing. (Afterwards the ArXiv version is updated.) I don't know of any reason it can't work for all of academia.
- RichardPrice 14y ago
  You're right that the cost isn't the main issue. It shouldn't be disregarded though: with 2 million articles published a year, a $2,000 cost per article means $4 billion a year that could be spent on more research. To put that in context, the NIH's budget for tackling cancer is $4.9 billion a year. The main issues with the journal system are the effects it has on the speed and quality of scientific communication. Peer review by two people is not robust, and it is very slow; secondly, the full range of scientific media aren't shared. These are the problems that, I believe, require an overhaul of the reputation system in science. The only way to incentivize scientists to share their comments on papers quickly, and at scale, is to provide rewards for them to do that. The range of media that scientists will start to share will be shared on platforms designed for that media type. Code will be shared on Github. Videos will be shared on YouTube. Data-sets will be shared on sites like Figshare or other sites dedicated to handling that content format well.

mjn 14y ago

Yes, I get that's the sales pitch, but you didn't address gamification at all, which is what you're promoting, and will positively harm science, not improve it. In addition, peer-review at well-run open access journals, which are the model I think we should be moving towards, does not take anywhere near 12 months, and in CS there are no publication fees (I gather this differs in biology). I typically get very well-thought-out reviews which help to improve my papers in about 6-8 weeks, and if accepted, publication is completely free of charge. And, it's becoming increasingly common to circulate preprints ahead of submission, on places such as the arXiv or at conferences or workshops, to get the feedback of a larger proportion of the community before publishing the final version of the paper. I agree peer review does not settle the correctness of a paper; it's a system for ensuring a high-quality discourse, not to publish only papers which are the last word on a subject. That's what reply papers, replication studies, etc. are for. I am skeptical that attaching a bunch of points and badges to the discourse is going to improve the quality of that discourse, and suspect it will harm it. I'd rather look at how to improve the discourse by improving the journals. But that won't make money for VCs, so I'm not surprised that we're only seeing it from nonprofit organizations like JAIR and PLoS.

RichardPrice 14y ago
With the time-lag, you have to take into account the rejection rate. Most articles don't get into their first choice journal, and perhaps get into their second or third choice. You can't parallel submit, meaning that the time-lag between finishing the paper, and it getting published in a journal, often ranges between 6 and 18 months. I understand that Computer Science does have a slightly faster peer review process than other fields. Here is a piece where a leading neuroscientist writes "Today the lag [in my field] between submission and acceptance is often more than a year". http://www.pnas.org/content/early/2013/01/25/201300924.short?rss=1 http://www.pnas.org/content/early/2013/01/25/201300924.short... Gamification is an important issue. You should note that the journal's peer review system, based on two people reading a paper, gets gamed. There is the wide-spread practice of defensive citation, where you defensively cite anyone who might conceivably peer review your paper, in order to maximize the chances of its getting accepted. Some people know how to play that game better than others. One way of handling gamification is to look at the quality of the people interacting with a piece of content, and not just the quantity. This is how Google's PageRank algorithm deals with the issue. There is also the approach, taken by Twitter and Facebook, of picking out the people's whose judgements you trust, and then letting them act as quality filters. Those systems can be gamed too, but aren't typically gamed to a large extent.
- jacquesm 14y ago
  I think you are confusing gamification and gaming the system, those are not the same thing. GP was referring to badges, points etc (gamification) and you are referring to gaming the system (ie citing defensively).
  - jessriedel 14y ago
    If so, then mjn needs to clearly distinguish between (bad) gamification and a (good) reputation system, which will necessarily involve something like points or certifications.
- notahacker 14y ago
  "Defensive" citations are arguably consistent with good research practice in that they reference all relevant research by the most prominent names in the field; it's certainly no worse than the sort of open trading of citations a PageRank style algorithm theoretically encourages. Sorry, I don't think Twitter and Facebook's quality filters "aren't typically gamed to a large extent" and I do think they're incomparably worse.

jessriedel 14y ago

> Open access journals, based on authors paying $1,000-$3,000 to publish a paper, are going to be a temporary stepping stone on the way to the decline of the journal. At least in my field, papers cost roughly $50k-$100k for just the researcher salary and overhead. Paying $2k to get a paper published is not a high hurdle (especially once the norm of closed-access publishing fades away). And this really seems like the minimum cost that needs to be incurred in order to get decent copyediting and the editorial cost of co-ordinating the referees. I love the ArXiv, but if I can get a the copyedited version from the journal I will. Yes the referees aren't paid, which may or may not change, but paying a PhD editor $500 to figure out the best referees and engage in the back-and-forth with them and the authors is pretty minimal. So $2k for a professionally edited, polished paper which is then free to everyone forever seems like a pretty good deal, and essentially a non-issue. Your point about the need for incentives for scientists to share non-paper scientific output like code is well-taken (and well-known). But I don't see why this can't be solved by open-access journals. I am completely unconvinced that facebook likes controlled by a for-profit organization are the way to go. EDIT: By the way, you've mentioned in several places the 12-month time lag for publishing. This was solved for the physics community by the ArXiv; the papers get immediately released, and you can still submit to journals for the benefits of copyediting and refereeing. (Afterwards the ArXiv version is updated.) I don't know of any reason it can't work for all of academia.

RichardPrice 14y ago
You're right that the cost isn't the main issue. It shouldn't be disregarded though: with 2 million articles published a year, a $2,000 cost per article means $4 billion a year that could be spent on more research. To put that in context, the NIH's budget for tackling cancer is $4.9 billion a year. The main issues with the journal system are the effects it has on the speed and quality of scientific communication. Peer review by two people is not robust, and it is very slow; secondly, the full range of scientific media aren't shared. These are the problems that, I believe, require an overhaul of the reputation system in science. The only way to incentivize scientists to share their comments on papers quickly, and at scale, is to provide rewards for them to do that. The range of media that scientists will start to share will be shared on platforms designed for that media type. Code will be shared on Github. Videos will be shared on YouTube. Data-sets will be shared on sites like Figshare or other sites dedicated to handling that content format well.