4 ms·
The balance is gone, at least in the US, because of several reasons, some easily quantifiable, others more subtle or difficult to quantify. Most of it is due t
by 0d9eooo 6y ago
The balance is gone, at least in the US, because of several reasons, some easily quantifiable, others more subtle or difficult to quantify.
Most of it is due to the prevalence of indirect fund systems on grants in the US. The way this is budgeted, universities bring in far more money than their actual costs, meaning there's huge incentives to prioritize grant funding over actual research. Because of huge competition and the way grants are evaluated, you have grants only weakly being related to impact or progress, and instead being targeted to what is most likely to be popular.
Another contributing factor is reliance on bibliometric indices like h indices etc. Although this seems reasonable, what it leads to is huge pressure to publish what's popular over other considerations, with lots of co-authors. I've reviewed for high-status journals have said explicitly in instructions to reviewers (and I'm only weakly paraphrasing) "don't evaluate the methodological integrity or integrity of the conclusions, only evaluate how popular this is likely to be."
What happens is you're incentivizing groupthink very heavily. Combine this with eroding of tenure (either through elimination of tenure-track lines, or protections of intellectual exploration), especially in places like biomedicine where the financial conflicts of interest are huge, it leads to big problems.
I have mixed feelings about peer review. I do think it's overrated, broken, and not transparent enough. Having done it for over a decade, I'm still being surprised by ethically questionable things that happen (ironically in my experience it seems like this actually happens the most in fields that are supposed to be more "objective", like statistics or math). On the other hand, my emerging experience with the unbridled internet is that it just reinforces the TED-ization of academics, in that things like twitter attention get mixed in with more formal evaluation of research.
Given the traditional peer review system versus some more open free-for-all, I'd choose the latter for multiple reasons, but they both have issues. In today's dysincentivised climate, the more open approach does have downsides, but maybe if the culture was fixed in other ways those concerns would be alleviated.
To address your question about what to do: there have been lots of proposals but they don't seem to get traction.
Open publication, archives, etc. seem to be moving in the right direction, but many of the grant solutions (eliminating indirect funds or making them line-item justified, lottery systems, funding based on publications through awards) haven't gotten traction.
- impendia 6y ago> I've reviewed for high-status journals have said explicitly in instructions to reviewers (and I'm only weakly paraphrasing) "don't evaluate the methodological integrity or integrity of the conclusions, only evaluate how popular this is likely to be." (Academic mathematician here) I'm curious, how do they get away with this? In mathematics, it's common and perfectly acceptable to turn down requests to review a paper. (You're not supposed to turn down all of them.) When I've done so, because I've already taken on too many reviewing assignments at one time, I've never gotten pushback from journal editors. If some journal editor asked me to review a paper, and gave me instructions along the lines that you described, then I'd just tell them "I'm very sorry, but I'm unusually busy at the moment."
- 0d9eooo 6y agoThose comments weren't at a math journal, but a more applied biomedical journal. I also have started refusing requests from that journal. The instructions I referred to werent available until well after accepting the review request, for what it's worth.
- impendia 6y agoThey give instructions only after you have accepted? That, too, seems odd to me. I guess that it is what it is.