3 ms·
Yes, I get that's the sales pitch, but you didn't address gamification at all, which is what you're promoting, and will positively harm science, not improve it.
by mjn 14y ago
Yes, I get that's the sales pitch, but you didn't address gamification at all, which is what you're promoting, and will positively harm science, not improve it.
In addition, peer-review at well-run open access journals, which are the model I think we should be moving towards, does not take anywhere near 12 months, and in CS there are no publication fees (I gather this differs in biology). I typically get very well-thought-out reviews which help to improve my papers in about 6-8 weeks, and if accepted, publication is completely free of charge. And, it's becoming increasingly common to circulate preprints ahead of submission, on places such as the arXiv or at conferences or workshops, to get the feedback of a larger proportion of the community before publishing the final version of the paper.
I agree peer review does not settle the correctness of a paper; it's a system for ensuring a high-quality discourse, not to publish only papers which are the last word on a subject. That's what reply papers, replication studies, etc. are for. I am skeptical that attaching a bunch of points and badges to the discourse is going to improve the quality of that discourse, and suspect it will harm it. I'd rather look at how to improve the discourse by improving the journals. But that won't make money for VCs, so I'm not surprised that we're only seeing it from nonprofit organizations like JAIR and PLoS.
- RichardPrice 14y agoWith the time-lag, you have to take into account the rejection rate. Most articles don't get into their first choice journal, and perhaps get into their second or third choice. You can't parallel submit, meaning that the time-lag between finishing the paper, and it getting published in a journal, often ranges between 6 and 18 months. I understand that Computer Science does have a slightly faster peer review process than other fields. Here is a piece where a leading neuroscientist writes "Today the lag [in my field] between submission and acceptance is often more than a year". http://www.pnas.org/content/early/2013/01/25/201300924.short?rss=1 http://www.pnas.org/content/early/2013/01/25/201300924.short... Gamification is an important issue. You should note that the journal's peer review system, based on two people reading a paper, gets gamed. There is the wide-spread practice of defensive citation, where you defensively cite anyone who might conceivably peer review your paper, in order to maximize the chances of its getting accepted. Some people know how to play that game better than others. One way of handling gamification is to look at the quality of the people interacting with a piece of content, and not just the quantity. This is how Google's PageRank algorithm deals with the issue. There is also the approach, taken by Twitter and Facebook, of picking out the people's whose judgements you trust, and then letting them act as quality filters. Those systems can be gamed too, but aren't typically gamed to a large extent.
- jacquesm 14y agoI think you are confusing gamification and gaming the system, those are not the same thing. GP was referring to badges, points etc (gamification) and you are referring to gaming the system (ie citing defensively).
- jessriedel 14y agoIf so, then mjn needs to clearly distinguish between (bad) gamification and a (good) reputation system, which will necessarily involve something like points or certifications.
- notahacker 14y ago"Defensive" citations are arguably consistent with good research practice in that they reference all relevant research by the most prominent names in the field; it's certainly no worse than the sort of open trading of citations a PageRank style algorithm theoretically encourages. Sorry, I don't think Twitter and Facebook's quality filters "aren't typically gamed to a large extent" and I do think they're incomparably worse.