3 ms·

Hard work doesn't scale. If you are already an expert in the field, you need a few months to properly evaluate someone's work. And if there are 100 candidates f

Hard work doesn't scale. If you are already an expert in the field, you need a few months to properly evaluate someone's work. And if there are 100 candidates for the position (which is pretty common), you have to spend decades evaluating them. In practice, you reject the vast majority of the candidates by reputation, publication venues, paper counts, grants and awards, and similar metrics. And then, if you are particularly interested, you may devote a few days on superficial reading of the works of the shortlisted candidates.

godelski 2y ago
> Hard work doesn't scale Not with the current framing but you contradict yourself. It's true that there are hundreds of candidates but that's... Scale. Getting a PhD is no easy task. And I'm not saying you should abandon the metrics all together, that would be a naive misinterpretation especially given what I made explicit. Instead consider the effort in evaluation should scale with the role. Maybe you can reject 80% of candidates by citations and indices alone. Do you miss some gold candidates? Probably, but it's fuzzy. Your remaining 20% you can consider their niches, funding capacity, and read at least the abstracts of their top works. But in your final selection you should be nuanced and read a selected list of their works (not all. Let the candidate tell you). Is it hard? Hell yeah. But why should something being hard stop us from doing it? That's called giving up. It's a crazy mentality to have as a researcher. > you may devote a few days on superficial reading of the works of the shortlisted candidates. So it's okay to do this with a paper when we review for publication but this is too much work when hiring someone who's going to be your colleague for years to come? Come on, I can't be the only one that thinks this is ludicrous.
- jltsiren 2y ago
  Evaluation by reputation and metrics is the norm. Evaluation by the quality of your work is the exception. You have to be particularly good by reputation and metrics in order to reach the stage where your work may be evaluated. And even when your work is evaluated, it's based on superficial reading. The people reading it (who are hopefully experts in the subfield) don't have the time to become experts in that specific niche to judge it properly. What they do is similar to peer review. And like all peer review, it's far more useful for spotting weak points and suggesting improvements than for ranking people/works.
  - godelski 2y ago
    > Evaluation by reputation and metrics is the norm. Evaluation by the quality of your work is the exception You don't see a problem with this? Because I sure do. > And even when your work is evaluated, it's based on superficial reading. The people reading it Ditto This is exactly what I'm complaining about and saying we should do better. Just because things suck doesn't mean you need to defend them. We can in fact change things.
    - jltsiren 2y ago
      I don't see a problem that could be solved within the current constraints. If we could convince taxpayers and private funders to double or triple the funding for academic research, many things could be done better. But today we have too many people trying to do a career in research and too little funding for them. Most issues in the academia arise from that.
      - godelski 2y ago
        Then look for ways to solve it. I clearly didn't present a full solution and I don't have one. But the biggest issue is people being apathetic or trying to say there is no problem[0]. We're researchers. We solve crazy hard problems all the time. I'm not saying our "house" needs to be completely in order, but right now it seems to be in dismay. I'm certain we can solve this. We put a fucking man on the moon, we found the Higgs and gravitational waves, we made computers talk, and we rid the world of deadly diseases. Surely we can solve a problem like this. If anything, those should be reasons to motivate us to solve this, so we can do more! But I do think a problem is how people interpret economics. Maybe a research paper doesn't directly lead to a trillion dollar company, but those companies certainly rely on a lot of publicly funded research. The returns on investment are pretty high and that's worthwhile. [0] I believe some feel the need to defend a system due to their success in it. This needs to be disentangled. A critique on the system is not a critique on "you" or "your work".

godelski 2y ago

> Hard work doesn't scale Not with the current framing but you contradict yourself. It's true that there are hundreds of candidates but that's... Scale. Getting a PhD is no easy task. And I'm not saying you should abandon the metrics all together, that would be a naive misinterpretation especially given what I made explicit. Instead consider the effort in evaluation should scale with the role. Maybe you can reject 80% of candidates by citations and indices alone. Do you miss some gold candidates? Probably, but it's fuzzy. Your remaining 20% you can consider their niches, funding capacity, and read at least the abstracts of their top works. But in your final selection you should be nuanced and read a selected list of their works (not all. Let the candidate tell you). Is it hard? Hell yeah. But why should something being hard stop us from doing it? That's called giving up. It's a crazy mentality to have as a researcher. > you may devote a few days on superficial reading of the works of the shortlisted candidates. So it's okay to do this with a paper when we review for publication but this is too much work when hiring someone who's going to be your colleague for years to come? Come on, I can't be the only one that thinks this is ludicrous.

jltsiren 2y ago
Evaluation by reputation and metrics is the norm. Evaluation by the quality of your work is the exception. You have to be particularly good by reputation and metrics in order to reach the stage where your work may be evaluated. And even when your work is evaluated, it's based on superficial reading. The people reading it (who are hopefully experts in the subfield) don't have the time to become experts in that specific niche to judge it properly. What they do is similar to peer review. And like all peer review, it's far more useful for spotting weak points and suggesting improvements than for ranking people/works.
- godelski 2y ago
  > Evaluation by reputation and metrics is the norm. Evaluation by the quality of your work is the exception You don't see a problem with this? Because I sure do. > And even when your work is evaluated, it's based on superficial reading. The people reading it Ditto This is exactly what I'm complaining about and saying we should do better. Just because things suck doesn't mean you need to defend them. We can in fact change things.
  - jltsiren 2y ago
    I don't see a problem that could be solved within the current constraints. If we could convince taxpayers and private funders to double or triple the funding for academic research, many things could be done better. But today we have too many people trying to do a career in research and too little funding for them. Most issues in the academia arise from that.
    - godelski 2y ago
      Then look for ways to solve it. I clearly didn't present a full solution and I don't have one. But the biggest issue is people being apathetic or trying to say there is no problem[0]. We're researchers. We solve crazy hard problems all the time. I'm not saying our "house" needs to be completely in order, but right now it seems to be in dismay. I'm certain we can solve this. We put a fucking man on the moon, we found the Higgs and gravitational waves, we made computers talk, and we rid the world of deadly diseases. Surely we can solve a problem like this. If anything, those should be reasons to motivate us to solve this, so we can do more! But I do think a problem is how people interpret economics. Maybe a research paper doesn't directly lead to a trillion dollar company, but those companies certainly rely on a lot of publicly funded research. The returns on investment are pretty high and that's worthwhile. [0] I believe some feel the need to defend a system due to their success in it. This needs to be disentangled. A critique on the system is not a critique on "you" or "your work".