3 ms·
I've been working full time on trying to help solve one of the problems in academia (access to research), and although I was superficially aware of most of the
by Vinnl 9y ago
I've been working full time on trying to help solve one of the problems in academia (access to research), and although I was superficially aware of most of the problems in academia, once you start to dive into one you will learn a lot about the others as well, as they're somewhat intertwined. And really, the scope and breadth of the problems is astonishing.
Personally, my hunch is that the root cause for most of them is a somewhat misguided focus on "excellence" combined with it being practically impossible to measure researcher performance with a reasonable amount of effort. Then again, the problems are so intertwined and have so many different actors, that finding the root cause is really hard, so who knows whether I'm correct.
- dalbasal 9y agoI think measuring research performance is a dead end. People can do this, although they carry a ton of blindness and blindspots. Bureaucracies, systems and such can't. They end up measuring what they can measure, like grades, attendance, number of published papers, number of citations maybe... It just won't end well. But, we can have systems that promote performance without measuring it. I think those are the kinds of ideas we should look at.
- Vinnl 9y agoI agree with your first paragraph, but I don't see how your second paragraph would work. Do you have any concrete examples/ideas of how such a system could work? (Without requiring e.g. those on tenure committees to read and understand all the research of all applications - unless you believe that could work somehow.)
- dalbasal 9y agoNothing specific. I'm also not an academic, so anything I say will be a little contrived but... ideas generally... I think the first step is look outside the basic, low-level structure and look a level or two above it. I think the best ideas will likely touch on a few of the related problems and happen outside of individual departments. Either the publication or the funding structures seem like good places to start. Publishing... This is obviously a centrepiece institution. It's central to replication/quality issues, to publish-or-perish career incentives and to funding allocation. So, I guess any radical ideas here will likely impact on a lot. Can we modernize publication, make it more open, and update the peer review system? One step is to move away from seeing "peer reviewed" as a binary. Another might be increasing the role of peer review. Maybe papers should have critical "reviewer notes." Maybe publications could take more of a 'director' role, influencing follow up, replications, being critical of practices... ..If a paper concludes that "more research is needed," does this mean anything? Will the research be done? What exactly needs to be done (eg, same study @ 5X the sample size). Is this likely to yield a result? Is that result important? A big part of the replication problem (from my perspective, safely on my couch) seems to be that very little replication is done, because the incentives are all wrong. If institutions were different (eg capitalism, for example) I could imagine the problem being reversed: replication and further work could be overinvested in, impoverishing exploratory science instead of the other way around. Replication is a lot more concrete (and money men love concrete).. you know what you need to do specifically, and what you might achieve by doing it. The problem in academia is that original work, even unreplicated an inconclusive is valued higher than subsequent work. We have to incentivize replication. In social sciences, we're sitting on a mountain of inconclusive results. Unless this work is done, we can't build on. Funding... Measuring researcher output well enough to base funding on is not possible, imo. But, maybe we'd have better luck with portfolios. Ie, structure funding bodies as competing portfolio managers, looking to impress based on portfolio performance. This might give us a better mix of predictable returns (eg replicate these 175 studies) and long tail risky stuff.
- Vinnl 9y agoThese are interesting thoughts, so thank you for sharing. Especially the idea that reversed incentives (for replication) might indeed not be desirable as well was insightful to me. I very much agree that it would be great if "peer reviewed" was not seen as a binary - because the research it represents is not binary. However, basing funding decisions on a lot of portfolio's full of non-binary research results is still, I'm afraid, too much work for e.g. tenure committees. (The annoying thing here is that just because a system like what you describe is still far from perfect, people will stick with the system that's in place even though that's far worse...)