3 ms·

Independent professional reviewers should exist in the academia. Currently it's those who do research also serve as reviewers, but these people are very busy in

Independent professional reviewers should exist in the academia. Currently it's those who do research also serve as reviewers, but these people are very busy in their own research and don't necessarily review others' papers carefully. It's a shame that most of the funding goes to researchers who perform novel research, not too much incentive for reproducing results and reviewing papers. This is as if the science is built upon an unstable foundation, and lots of man- hours and powers will be wasted from that.

btrettel 8y ago
I agree wholeheartedly. During my PhD, I did a big data compilation in my field (the fluid dynamics of certain spray nozzles), which ended up convincing me many commonly believed things in the field are false. I'm currently planning at least 3 paper on the issues I've found. This took a lot of time, but ultimately would save even more time and money if people were willing to fund it. I did the work while I was a teaching assistant and doubt many funding agencies would be willing to fund someone for a semester on this. I'd like to be a professional reviewer. The closest existing job is a patent examiner, but they incentives there aren't great either. After my PhD I intend to at least do some consulting on the side, and perhaps "professional reviewer" could be a service I could offer. Worst case is that no one wants the service.
_delirium 8y ago
In this case the thing being criticized is a book, which is a somewhat different case. Reputable academic presses do send out book manuscripts for review, but it's a very different process from paper reviews, often involving only a partial early manuscript. I don't know how every press does it, but when I've reviewed for an academic press, I was given a draft of the first three chapters, and asked to give a recommendation for whether this was a serious academic book worth publishing, along with my positive and negative comments on the partial draft. There are obviously some things I could usefully do there: it's often possible to identify and reject total crank books, to notice some early tendency towards overstating claims, to give general feedback on the direction of the book and whether it is likely to reach its audience, etc. But it's not the same as a journal, where you have the full paper to review and are expected to read it and give detailed feedback (which is also a more reasonable thing to ask of reviewers, because the text isn't as long!). This reviewing approach is pretty traditional (it's how most famous philosophy books were published, for example) and relies heavily on good faith. Reviewers are supposed to judge whether this is a serious historian writing a good-faith book, but if they say yes, then you let the historian publish the book they want to publish without a lot of further rounds of vetting. It isn't expected they'll get everything right, but it's expected they'll make an earnest effort to get it right, perhaps even by employing research assistants themselves to do fact-checking. Anything they get wrong can then be corrected in the literature by someone else writing their own article or book in response. One exception are books which are adaptations of dissertations. In that case there would have been more substantial review of the claims, by the dissertation committee, assuming nothing fishy happens between dissertation acceptance and adaptation into a book. The first author of the paper linked here (Lieberman) has a book out on the history of sex toys that's an adaptation of her dissertation, and tells a pretty different history than the one she criticizes in this paper. The book being criticized (by Maines) is not a dissertation adaptation, though.

btrettel 8y ago

I agree wholeheartedly. During my PhD, I did a big data compilation in my field (the fluid dynamics of certain spray nozzles), which ended up convincing me many commonly believed things in the field are false. I'm currently planning at least 3 paper on the issues I've found. This took a lot of time, but ultimately would save even more time and money if people were willing to fund it. I did the work while I was a teaching assistant and doubt many funding agencies would be willing to fund someone for a semester on this. I'd like to be a professional reviewer. The closest existing job is a patent examiner, but they incentives there aren't great either. After my PhD I intend to at least do some consulting on the side, and perhaps "professional reviewer" could be a service I could offer. Worst case is that no one wants the service.

_delirium 8y ago

In this case the thing being criticized is a book, which is a somewhat different case. Reputable academic presses do send out book manuscripts for review, but it's a very different process from paper reviews, often involving only a partial early manuscript. I don't know how every press does it, but when I've reviewed for an academic press, I was given a draft of the first three chapters, and asked to give a recommendation for whether this was a serious academic book worth publishing, along with my positive and negative comments on the partial draft. There are obviously some things I could usefully do there: it's often possible to identify and reject total crank books, to notice some early tendency towards overstating claims, to give general feedback on the direction of the book and whether it is likely to reach its audience, etc. But it's not the same as a journal, where you have the full paper to review and are expected to read it and give detailed feedback (which is also a more reasonable thing to ask of reviewers, because the text isn't as long!). This reviewing approach is pretty traditional (it's how most famous philosophy books were published, for example) and relies heavily on good faith. Reviewers are supposed to judge whether this is a serious historian writing a good-faith book, but if they say yes, then you let the historian publish the book they want to publish without a lot of further rounds of vetting. It isn't expected they'll get everything right, but it's expected they'll make an earnest effort to get it right, perhaps even by employing research assistants themselves to do fact-checking. Anything they get wrong can then be corrected in the literature by someone else writing their own article or book in response. One exception are books which are adaptations of dissertations. In that case there would have been more substantial review of the claims, by the dissertation committee, assuming nothing fishy happens between dissertation acceptance and adaptation into a book. The first author of the paper linked here (Lieberman) has a book out on the history of sex toys that's an adaptation of her dissertation, and tells a pretty different history than the one she criticizes in this paper. The book being criticized (by Maines) is not a dissertation adaptation, though.