3 ms·

Doing science in the open

6 points by robg 17y ago

lutorm 17y ago
There are many things that make the analogies made in the article not applicable. If I buy a Pokemon, I can crank out a review in probably 5 min after playing with it for a while. Contrast that to peer review: to write a careful review of an article takes me at least a full work day to make sure that I understand what's done, think about possible pitfalls and weaknesses, and then write a comprehensive and helpful review. No one will put in that amount of time to write a comment on a web page, so the only people that can offer substantial comments would be those that have taken the time to understand and think about the paper for other reasons (usually because it directly concerns their own work). The problem is then that even if these few people actually make comments, they are likely to be totally drowned out by people who make useless comments (either because they don't understand the paper or because they have no incentive to make an effort). Regarding the shoe store, that's also a false analogy. A shoe store selling shoes is akin to a scientist publicizing their finished papers, they have nothing to lose by revealing their design because they already have them in the store. A better analogy would be to ask: if Nike's having some problem figuring out how to manufacture a shoe, do they ask Reebok for advice? Since Reebok works in the same field, they likely would have useful advice to give, but they don't because the danger of Reebok taking the design and running with it is too large. That's how it would be with scientists openly sharing their works in progress. I'm not debating that the progress of science would be helped with more open sharing, but offering questionable analogies does not help in understanding the problem. However, the problem of conflicting incentives is a very real one. As long as scientists are judged based on the numbers of published papers and numbers of citations, there's an incentive to hoarding data to squeeze every last possible paper out of them. Also, there really is little incentive to share tools with the community apart from gaining recognition. This is compounded by people being bad at citing papers describing tools, too, so you don't even get that reward.

6 points by robg 17y ago

There are many things that make the analogies made in the article not applicable. If I buy a Pokemon, I can crank out a review in probably 5 min after playing with it for a while. Contrast that to peer review: to write a careful review of an article takes me at least a full work day to make sure that I understand what's done, think about possible pitfalls and weaknesses, and then write a comprehensive and helpful review. No one will put in that amount of time to write a comment on a web page, so the only people that can offer substantial comments would be those that have taken the time to understand and think about the paper for other reasons (usually because it directly concerns their own work). The problem is then that even if these few people actually make comments, they are likely to be totally drowned out by people who make useless comments (either because they don't understand the paper or because they have no incentive to make an effort). Regarding the shoe store, that's also a false analogy. A shoe store selling shoes is akin to a scientist publicizing their finished papers, they have nothing to lose by revealing their design because they already have them in the store. A better analogy would be to ask: if Nike's having some problem figuring out how to manufacture a shoe, do they ask Reebok for advice? Since Reebok works in the same field, they likely would have useful advice to give, but they don't because the danger of Reebok taking the design and running with it is too large. That's how it would be with scientists openly sharing their works in progress. I'm not debating that the progress of science would be helped with more open sharing, but offering questionable analogies does not help in understanding the problem. However, the problem of conflicting incentives is a very real one. As long as scientists are judged based on the numbers of published papers and numbers of citations, there's an incentive to hoarding data to squeeze every last possible paper out of them. Also, there really is little incentive to share tools with the community apart from gaining recognition. This is compounded by people being bad at citing papers describing tools, too, so you don't even get that reward.