3 ms·

The same constraints you cite here applied to "peer review" articles. The academics doing the peer reviewing are also "busy with work" and want to do their own

The same constraints you cite here applied to "peer review" articles. The academics doing the peer reviewing are also "busy with work" and want to do their own thing, and especially in cases where reviewers are anonymous, the incentives facing the academics charged with the review are absolutely horrendous. My point with the part about the bot that checks errors is that, if we are being honest, in far too many cases that "peer review" is less than ironclad, to put it mildly. In that scenario, the journals actually do us a disservice because they attempt to signal that the content you are reading went through a strict bit of scrutiny before it reached your eyeballs. That is a dangerous assumption to believe in if it turns out that it really wasn't.

ylem 10y ago
I guess it depends on the size and culture of a given research community. Researchers are people too and petty things do happen. But, as a referee, I have followed through error propagation (once I saw erorbars that looked amazing and wondered how I could get such beautiful results and discovered they made an error in the error propagation). When reviewing methods papers, I have checked the integrals and equations, looking for errors (including minus signs) because eventually it will go into some code that most people won't read. Then, there are just simple checks on the science and on history of the field. I don't think that I'm unique. Refereeing well takes time, but it's part of the responsibility/duty of being a researcher. A friend's metric is that he should referee at least as many papers per year as he publishes. By all means, when you read a paper, be skeptical and go through it yourself, but in my field, referees do actually provide some quality control.
- ChicagoBoy11 10y ago
  This is as it should be. But the reader of the article has no idea whether you looked at it or someone with a far less significant commitment to the task than you. And what's keeping you and your friend doing this time-consuming work is your commitment to science and your sense of ethics and responsibility, which your community certainly does not universally and uniformly share. That commitment is also the first to go when career, money, lack of time, etc. start coming into the picture, or the pressure to abandon it is certainly high. That's what we need to find a way to alter: If you are doing great science -- including the painstaking work of carefully reading a paper and following minus signs through calculation and checking integrals and whatnot -- that should be explicitly rewarded in the profession, and shouldn't just rely on intrinsic values a researcher holds in order to survive. I don't see journals today being the leaders in driving that innovation.

ylem 10y ago

I guess it depends on the size and culture of a given research community. Researchers are people too and petty things do happen. But, as a referee, I have followed through error propagation (once I saw erorbars that looked amazing and wondered how I could get such beautiful results and discovered they made an error in the error propagation). When reviewing methods papers, I have checked the integrals and equations, looking for errors (including minus signs) because eventually it will go into some code that most people won't read. Then, there are just simple checks on the science and on history of the field. I don't think that I'm unique. Refereeing well takes time, but it's part of the responsibility/duty of being a researcher. A friend's metric is that he should referee at least as many papers per year as he publishes. By all means, when you read a paper, be skeptical and go through it yourself, but in my field, referees do actually provide some quality control.

ChicagoBoy11 10y ago
This is as it should be. But the reader of the article has no idea whether you looked at it or someone with a far less significant commitment to the task than you. And what's keeping you and your friend doing this time-consuming work is your commitment to science and your sense of ethics and responsibility, which your community certainly does not universally and uniformly share. That commitment is also the first to go when career, money, lack of time, etc. start coming into the picture, or the pressure to abandon it is certainly high. That's what we need to find a way to alter: If you are doing great science -- including the painstaking work of carefully reading a paper and following minus signs through calculation and checking integrals and whatnot -- that should be explicitly rewarded in the profession, and shouldn't just rely on intrinsic values a researcher holds in order to survive. I don't see journals today being the leaders in driving that innovation.