3 ms·
In fairness, all systems are broken, academia isn't much more broken than anything else involving large numbers of mortals. The basic problem is that we need t
by hugh4 11y ago
In fairness, all systems are broken, academia isn't much more broken than anything else involving large numbers of mortals.
The basic problem is that we need to rank one researcher's merit against everyone else's, and there's no perfect way to do this because nobody can familiarise themselves with every individual's research. Rely on peers' opinions to measure merit and you just reward back-scratching. Rely on objective measures like citation counts and h-index and you incentivise people to focus on these instead of research quality.
The thing is that macro-scale works pretty well. As an overall system, academic science produces new knowledge at an astounding rate, and in terms of dollars in to research produced I'd say it works pretty efficiently. The measures we use are correlated with research quality even if they don't measure it perfectly.
It's only at the detailed scale where it looks badly flawed.
edit: and of course whenever I spend a long time writing a thoughtful comment I get the "you're submitting too fast, please slow down" message. Time to leave this comment to sit for an hour before pressing reply again...
- amelius 11y ago> Rely on peers' opinions to measure merit and you just reward back-scratching. Rely on objective measures like citation counts and h-index and you incentivise people to focus on these instead of research quality. In the meantime, Google seems to have developed a pretty good ranking system that is difficult to game.
- grkvlt 11y agoDon't citation indexes work pretty much exactly like Google's PageRank, with cites taking the place of href links? A parent comment seems to suggest the same thing, so it looks like the ranking is gameable. In practice I understand this happens via citation-swapping (similar to link swapping, or paid links in blogs) and poor-quality journals accepting articles for a fee, or without review (much like paid content, or content farms) and papers that cite other articles just for the increase in rankings (similar to link farms) so the analogy to SEO (in its various hats) seems to fit. This would suggest that the only way to fix the 'broken' citation ranking system is the same as Google, and bolt on empirically determined heuristics above the pure mathematical model of PageRank, and black-list or penalise certain journals or types of citation?
- amelius 11y ago> and bolt on empirically determined heuristics above the pure mathematical model of PageRank The impact of Google doing this is, in financial terms, probably much larger than if the same type of approach would be applied in the academic situation. So if Google is allowed to do this, why can't we allow the same approach for research? Apparently, the system works quite well.
- tonypace 11y agoHonestly, funding a research team to develop a science ranking engine that was both being continually developed to thwart gaming and effectively independent of university and science-funding politics would be a truly excellent use of YCombinator's money.
- effie 11y agoIndeed. Simple link counting based search algorithms got gamed a lot in the past, by regular website operators and most heavily by spammers. That's partially why the current search algorithms are much more clever and secret than just using the Page rank. Human raters and machine learning off the data they generate are used as well. This helps to make the rules more uncertain and nasty SEO tricks much harder to discover and use. Compare this to the situation in rating of academic research, where the papers are rated by the community of authors itself and the ranking is essentially a weighted counting of references. Of course it leads to lots of self-serving spam. If the idea of independent raters (and possibly machine learning on top of it) could be implemented for academic papers as well, that would be a big improvement, I think. Academia has got a lot to learn from search engines.
- p4bl0 11y agoThe problem is not that it is gameable or not. Even if it is, most researchers won't try to game it. The problem is that they will focus on optimizing these counters by necessity (they have no choices, it is that or goodbye), rather than doing what they want to do: great research. I wouldn't call prioritizing quantity over quality gaming the ranking system, it is merely playing by its rules. I would call gaming it if there was some cheat (like you cite me and I cite you even if really we shouldn't), which may happen but I believe is rare.
- amelius 11y agoIt seems you have reduced my argument to the last part only. Google's system is pretty good even if you don't focus on gaming. By the way, even when it comes to websites, most users (admins) don't try to game the system.
- BurningFrog 11y ago> As an overall system, academic science produces new knowledge at an astounding rate Maybe. But I can't really imagine how you'd measure that, or what you'd compare it to. Even if the current system is pretty great, it can only get better with competition from alternative systems, like what's being announced here.
- eli_gottlieb 11y ago>As an overall system, academic science produces new knowledge at an astounding rate, and in terms of dollars in to research produced I'd say it works pretty efficiently. I wouldn't call throwing out your trained experts and starting with fresh trainees in a five-to-seven year cycle all that efficient.
- hugh4 11y agoIt is if you can get away with paying 'em twenty five grand a year for the duration. Kaching!