3 ms·
Let's think about academia for a second and the metrics we use. You must publish often and your work most be "novel." The more novel, the more prestige, the mor
by godelski 3y ago
Let's think about academia for a second and the metrics we use. You must publish often and your work most be "novel." The more novel, the more prestige, the more frequent the faster you rise.
This is a system that's problematic for science, but crazy in humanities. One reason in science is it discourages replication. But there is a parallel here in something like history, reinvestigation. Something like the Roman Empire is well studied and there's lower novelty to be gained. But do we want to discourage such research? Discourage modern people for maintaining such records?
Under the current system, we can encourage these people to continue studying popular areas but do we encourage maintaining historical truth and integrity? I conjecture that the answer is no. It encourages making things up, especially to fit a modern narrative. It's a system that makes people not know they're lying because they are true believers and no credible experts push back.
This isn't a problem unique to the study of history, but you can find issues like this is all research in academia. It is very hard to measure the quality of research. Citations, h-index, and so on do not mean a lot. We've seen in the past few months very popular researchers found to be frauds, with high positions in prestigious institution like Stanford. Even in practical areas that should be treatable: psychology, medicine, machine learning, and more. We want to believe, but do we have strong evidence and strong understanding? We make easy metrics at first to be "good enough" or because we lack better ones. But we often forget the limitations of those metrics as time goes by because we get caught up in the rat race. You can't just ask yourself if the metric makes sense, but instead need to ask if there's an easier way to score on the metric that isn't the intended object of measurement. We can see here that citations can be hacked easily by fun stories, surprising results, and we'll never uncover them because the only people who attempt to replicate are those in junior positions like undergrads and grad students, who are more likely to question their results than the original work. Those failures never get written about.
Science is really about progressing knowledge. Publications are really about communication. Both these are very vague and can be accomplished in many different ways.
I say instead of making narrow easy to define metrics that will never be good enough we embrace the chaos and noise. Let the researchers be free and let ̶G̶o̶d̶ time sort it out. We don't know what is good research until 5-20 years down the line. We don't know who's a good teacher until the same. By removing the metrics you force nuance to not be ignored. Most of the time it's a "know it when you see it" definition and if that's true we have to be careful about levels of confidence. The thing that makes us human is the ability to handle nuance. It is what differentiates us from animals and LLMs. If the downside is that in a field where people are required to think hard and carefully have to... Think hard and carefully, then that's okay. ML is very successful with the arxiv model because despite much hype, noise, and even many bad actors, us researchers can sniff it out. We have to do it anyways, as that's our job.
I fear if we don't fix things we will continue to just degrade our foundation. We've seen this experiment fail over the last century. It's time to just get rid of the bureaucracy. Let academics be academics. Micromanagement isn't helping. Research can take decades, but people concerned with the next quarter will never allow that. Academics is about risks, challenging the status quo, and nuance. It is not to be run like a business. I am happy for this sector to not be profitable because I know the profits are indirect and benefit everyone. But returns on investments may take centuries, and that's okay.
- sideshowb 3y agoI often think the same, in fact in my todo list is (in the background) think of more ways to reduce the bureaucracy I face on a daily basis, which takes many forms - from arguing with reviewer #2 over whether or not my finding is notable, to trying to convince a panel that my idea is worth funding. From a system design perspective, alas resources are limited and have to be allocated somehow, and in the absence of any metrics at all as you suggest, it would ultimately come back to who can tell the better story about their work. This is still at best only weakly correlated with the actual quality of the work. And so we run up against https://en.wikipedia.org/wiki/Jerry_Pournelle#Pournelle's_iron_law_of_bureaucracy https://en.wikipedia.org/wiki/Jerry_Pournelle#Pournelle's_ir... The situation could definitely be better than the current one, but alas I don't have any magic solution.
- godelski 3y agoExactly. But I think if we at least recognized that metrics are guides rather than targets we could get reviewer 2 to stop wasting everyone's time or get an AC to tell them to shut up. It wastes a lot of tax payers money. Similarly I think we need to recognize academic funding as that of long term investments. You can be highly risky and should be. In reality our total investment into this sector is very low, we aren't even investing a penny from our dollar. And I could go on about the scam of "peer reviewer" which just seems like a way to extract government money, using another players workforce, and providing at best a trivial amount of utility
- nvm0n2 3y agoThis is such an academic take! There's tons of fraud, bias and false claims in research, therefore the fix is to get rid of whatever trivial checks and balances we already have and let researchers do or say whatever they want. The "let the researchers be free" is the system we already have and it's not working. Researchers need much less freedom than they have today. They need to be micromanaged far more aggressively, by people who aren't impressed by them. Society needs to treat them as potential adversaries, regulate the hell out of them and toss them in prison when they step out of line, no different to company CEOs or corporate scientists (see Theranos). Also reducing their number by 20x or so would be a good start. Whatever happens, they definitely need to lose the freedom to publish false claims. Also universities are run by people who will never do this. Academic fraud happens all the time and there's never any follow up. Therefore universities need to lose their freedom too. There are really only two solutions to these sorts of frauds: 1. Defund it all, let the private sector seek the right level of management 2. Try to fix it with more micromanagement and regulation in the public sector. "Let academics be academics" is exactly how large sections of the population have ended up holding academia in outright contempt.