3 ms·
Nothing specific. I'm also not an academic, so anything I say will be a little contrived but... ideas generally... I think the first step is look outside the ba
by dalbasal 9y ago
Nothing specific. I'm also not an academic, so anything I say will be a little contrived but... ideas generally... I think the first step is look outside the basic, low-level structure and look a level or two above it. I think the best ideas will likely touch on a few of the related problems and happen outside of individual departments. Either the publication or the funding structures seem like good places to start.
Publishing... This is obviously a centrepiece institution. It's central to replication/quality issues, to publish-or-perish career incentives and to funding allocation. So, I guess any radical ideas here will likely impact on a lot.
Can we modernize publication, make it more open, and update the peer review system? One step is to move away from seeing "peer reviewed" as a binary. Another might be increasing the role of peer review. Maybe papers should have critical "reviewer notes."
Maybe publications could take more of a 'director' role, influencing follow up, replications, being critical of practices...
..If a paper concludes that "more research is needed," does this mean anything? Will the research be done? What exactly needs to be done (eg, same study @ 5X the sample size). Is this likely to yield a result? Is that result important?
A big part of the replication problem (from my perspective, safely on my couch) seems to be that very little replication is done, because the incentives are all wrong.
If institutions were different (eg capitalism, for example) I could imagine the problem being reversed: replication and further work could be overinvested in, impoverishing exploratory science instead of the other way around. Replication is a lot more concrete (and money men love concrete).. you know what you need to do specifically, and what you might achieve by doing it. The problem in academia is that original work, even unreplicated an inconclusive is valued higher than subsequent work. We have to incentivize replication. In social sciences, we're sitting on a mountain of inconclusive results. Unless this work is done, we can't build on.
Funding... Measuring researcher output well enough to base funding on is not possible, imo. But, maybe we'd have better luck with portfolios. Ie, structure funding bodies as competing portfolio managers, looking to impress based on portfolio performance. This might give us a better mix of predictable returns (eg replicate these 175 studies) and long tail risky stuff.
- Vinnl 9y agoThese are interesting thoughts, so thank you for sharing. Especially the idea that reversed incentives (for replication) might indeed not be desirable as well was insightful to me. I very much agree that it would be great if "peer reviewed" was not seen as a binary - because the research it represents is not binary. However, basing funding decisions on a lot of portfolio's full of non-binary research results is still, I'm afraid, too much work for e.g. tenure committees. (The annoying thing here is that just because a system like what you describe is still far from perfect, people will stick with the system that's in place even though that's far worse...)