5 ms·

> If you're hiring in your own field (which you should), you don't have to read the papers. You're already familiar with most of them, and judging by the venue

> If you're hiring in your own field (which you should), you don't have to read the papers. You're already familiar with most of them, and judging by the venue their published in, you also know their "level", so to speak. You're judging by reputation rather than substance. That opens up the possibility of manipulating reputation to achieve career goals instead of doing the hard work.

Ar-Curunir 2y ago
Unsurprisingly, in the vast majority of cases, substance and reputation tend to be at least loosely correlated, at least in academia. There are high-profile exceptions, but they are only high-profile because they are indeed exceptional.
- beedeebeedee 2y ago
  Perhaps, but it not only deviates from the practice of science but creates the very conditions that science has been developed to overcome- i.e., social status is accepted as the driver of research rather than the pursuit of knowledge. It might be considered ok for financial institutions to make decisions on who to fund or partner with based on status signals, but it is galling to think that is how our scientific institutions work. That practice seems as scientific as scientology.
kergonath 2y ago
> You're judging by reputation rather than substance. Is there an alternative, though? When hiring someone we use their degrees, experience and credentials, all of which are indirect ways of estimating competence. We cannot realistically expect people to know all about someone’s career when interviewing them, it’s not realistic. It’s the same thing for careers evolution. HR people don’t have time or skills to know about the science, and good scientists already spend way too much time reviewing articles or grant proposals, and sitting in various boards and panels. There is no need for more administrative work. I don’t think the current systems are ideal, but I think improving it would by fixing incentives would be more efficient (and that’s not only for scientists, it’s the same for any major organisation or company).
- beedeebeedee 2y ago
  @kergonath and @jitsiren in the sibling comment. You're both describing the outcome of a system of vastly disproportionate numbers of applicants and opportunities. I believe we should continue churning out intellectuals who are passionate about the sciences, but we need to provide them with enough opportunities, so there isn't the pressure for it to be gamed (because there is only opportunity for a few). Ultimately, if we don't do anything about it, it makes this problem just seem like the neglected by-product of thinly veiled money-laundering from the government to financial institutions (via university administrators and student loans).
- godelski 2y ago
  > Is there an alternative, though? Yes I said "read the god damn papers" but this is a condensed version of saying be more nuanced and familiar with the person's work. Metrics are signals, not answers. They are always fuzzy and we need to recognize that and bring this into the core of how we evaluate. You'll never be perfect and while we know to not let perfection stand in the way of good enough neither should you let lack of global optima stand in the way of progress. There's always room to improve. At the heart of academia we push human knowledge forward, even if just by little nudges at a time. But this means the environment changes as we succeed. What worked in the past may not work moving forward. And as we all should be intimately familiar with, given enough time Goodhart's Law always wins. But there's so many problems with becoming over reliant upon metrics. It's not a static measurement, it is a degrading one. This is because we get lost seeking success for ourselves because doing so makes it easier to do the work we want to (and ideally more impact on humanity). Because of this the best way to advance your career is not to continue researching but to grow the size of your lab and get grad students and post docs to research. You need to become a PI, who's job is political and about funding. This person does need a strong PhD background but it's also silly that it's extremely hard to find a path where your primary job after getting a PhD is... Research... But pushing the edge of what we know is fucking hard. It should be no surprise that it is hard to evaluate the performance of people trying to do this. So forgive me if I think in addition to using metrics like citations, venue pubs, h-indexes, you can take the time to read you colleagues papers. I'm really not asking for much here. One of the things I also find rotten in academia is this isolation. It's weird that we don't know, intimately, the work of those in our department. We should be reading their papers in the first place! I'm glad academia encouraged collaboration with groups outside your institution, but it shouldn't come at the cost of collaboration in house. Both are valuable. There's just so much more and I think it's why many are willing to tear down the system and start over. I'd rather not, but things are too broken for patchwork. We don't need to tear it down, but we do need to do some serious renovation. I encourage everyone to think deeply about the problem, even if you don't think there is one (especially if you don't!). Address the underlying assumptions that no one says. Consider when they hold and when they don't. Then ask if this works. If you believe so, great! I'd love to hear that opinion. But I find it absurd we can't spend a little time questioning the vehicle we are all riding in. It should be questioned frequently and regularly. Because maintenance is far easier than fixing something when it's broken.
jltsiren 2y ago
Hard work doesn't scale. If you are already an expert in the field, you need a few months to properly evaluate someone's work. And if there are 100 candidates for the position (which is pretty common), you have to spend decades evaluating them. In practice, you reject the vast majority of the candidates by reputation, publication venues, paper counts, grants and awards, and similar metrics. And then, if you are particularly interested, you may devote a few days on superficial reading of the works of the shortlisted candidates.
- godelski 2y ago
  > Hard work doesn't scale Not with the current framing but you contradict yourself. It's true that there are hundreds of candidates but that's... Scale. Getting a PhD is no easy task. And I'm not saying you should abandon the metrics all together, that would be a naive misinterpretation especially given what I made explicit. Instead consider the effort in evaluation should scale with the role. Maybe you can reject 80% of candidates by citations and indices alone. Do you miss some gold candidates? Probably, but it's fuzzy. Your remaining 20% you can consider their niches, funding capacity, and read at least the abstracts of their top works. But in your final selection you should be nuanced and read a selected list of their works (not all. Let the candidate tell you). Is it hard? Hell yeah. But why should something being hard stop us from doing it? That's called giving up. It's a crazy mentality to have as a researcher. > you may devote a few days on superficial reading of the works of the shortlisted candidates. So it's okay to do this with a paper when we review for publication but this is too much work when hiring someone who's going to be your colleague for years to come? Come on, I can't be the only one that thinks this is ludicrous.
  - jltsiren 2y ago
    Evaluation by reputation and metrics is the norm. Evaluation by the quality of your work is the exception. You have to be particularly good by reputation and metrics in order to reach the stage where your work may be evaluated. And even when your work is evaluated, it's based on superficial reading. The people reading it (who are hopefully experts in the subfield) don't have the time to become experts in that specific niche to judge it properly. What they do is similar to peer review. And like all peer review, it's far more useful for spotting weak points and suggesting improvements than for ranking people/works.

Ar-Curunir 2y ago

Unsurprisingly, in the vast majority of cases, substance and reputation tend to be at least loosely correlated, at least in academia. There are high-profile exceptions, but they are only high-profile because they are indeed exceptional.

beedeebeedee 2y ago
Perhaps, but it not only deviates from the practice of science but creates the very conditions that science has been developed to overcome- i.e., social status is accepted as the driver of research rather than the pursuit of knowledge. It might be considered ok for financial institutions to make decisions on who to fund or partner with based on status signals, but it is galling to think that is how our scientific institutions work. That practice seems as scientific as scientology.

kergonath 2y ago

> You're judging by reputation rather than substance. Is there an alternative, though? When hiring someone we use their degrees, experience and credentials, all of which are indirect ways of estimating competence. We cannot realistically expect people to know all about someone’s career when interviewing them, it’s not realistic. It’s the same thing for careers evolution. HR people don’t have time or skills to know about the science, and good scientists already spend way too much time reviewing articles or grant proposals, and sitting in various boards and panels. There is no need for more administrative work. I don’t think the current systems are ideal, but I think improving it would by fixing incentives would be more efficient (and that’s not only for scientists, it’s the same for any major organisation or company).

beedeebeedee 2y ago
@kergonath and @jitsiren in the sibling comment. You're both describing the outcome of a system of vastly disproportionate numbers of applicants and opportunities. I believe we should continue churning out intellectuals who are passionate about the sciences, but we need to provide them with enough opportunities, so there isn't the pressure for it to be gamed (because there is only opportunity for a few). Ultimately, if we don't do anything about it, it makes this problem just seem like the neglected by-product of thinly veiled money-laundering from the government to financial institutions (via university administrators and student loans).
godelski 2y ago
> Is there an alternative, though? Yes I said "read the god damn papers" but this is a condensed version of saying be more nuanced and familiar with the person's work. Metrics are signals, not answers. They are always fuzzy and we need to recognize that and bring this into the core of how we evaluate. You'll never be perfect and while we know to not let perfection stand in the way of good enough neither should you let lack of global optima stand in the way of progress. There's always room to improve. At the heart of academia we push human knowledge forward, even if just by little nudges at a time. But this means the environment changes as we succeed. What worked in the past may not work moving forward. And as we all should be intimately familiar with, given enough time Goodhart's Law always wins. But there's so many problems with becoming over reliant upon metrics. It's not a static measurement, it is a degrading one. This is because we get lost seeking success for ourselves because doing so makes it easier to do the work we want to (and ideally more impact on humanity). Because of this the best way to advance your career is not to continue researching but to grow the size of your lab and get grad students and post docs to research. You need to become a PI, who's job is political and about funding. This person does need a strong PhD background but it's also silly that it's extremely hard to find a path where your primary job after getting a PhD is... Research... But pushing the edge of what we know is fucking hard. It should be no surprise that it is hard to evaluate the performance of people trying to do this. So forgive me if I think in addition to using metrics like citations, venue pubs, h-indexes, you can take the time to read you colleagues papers. I'm really not asking for much here. One of the things I also find rotten in academia is this isolation. It's weird that we don't know, intimately, the work of those in our department. We should be reading their papers in the first place! I'm glad academia encouraged collaboration with groups outside your institution, but it shouldn't come at the cost of collaboration in house. Both are valuable. There's just so much more and I think it's why many are willing to tear down the system and start over. I'd rather not, but things are too broken for patchwork. We don't need to tear it down, but we do need to do some serious renovation. I encourage everyone to think deeply about the problem, even if you don't think there is one (especially if you don't!). Address the underlying assumptions that no one says. Consider when they hold and when they don't. Then ask if this works. If you believe so, great! I'd love to hear that opinion. But I find it absurd we can't spend a little time questioning the vehicle we are all riding in. It should be questioned frequently and regularly. Because maintenance is far easier than fixing something when it's broken.

jltsiren 2y ago

Hard work doesn't scale. If you are already an expert in the field, you need a few months to properly evaluate someone's work. And if there are 100 candidates for the position (which is pretty common), you have to spend decades evaluating them. In practice, you reject the vast majority of the candidates by reputation, publication venues, paper counts, grants and awards, and similar metrics. And then, if you are particularly interested, you may devote a few days on superficial reading of the works of the shortlisted candidates.

godelski 2y ago
> Hard work doesn't scale Not with the current framing but you contradict yourself. It's true that there are hundreds of candidates but that's... Scale. Getting a PhD is no easy task. And I'm not saying you should abandon the metrics all together, that would be a naive misinterpretation especially given what I made explicit. Instead consider the effort in evaluation should scale with the role. Maybe you can reject 80% of candidates by citations and indices alone. Do you miss some gold candidates? Probably, but it's fuzzy. Your remaining 20% you can consider their niches, funding capacity, and read at least the abstracts of their top works. But in your final selection you should be nuanced and read a selected list of their works (not all. Let the candidate tell you). Is it hard? Hell yeah. But why should something being hard stop us from doing it? That's called giving up. It's a crazy mentality to have as a researcher. > you may devote a few days on superficial reading of the works of the shortlisted candidates. So it's okay to do this with a paper when we review for publication but this is too much work when hiring someone who's going to be your colleague for years to come? Come on, I can't be the only one that thinks this is ludicrous.
- jltsiren 2y ago
  Evaluation by reputation and metrics is the norm. Evaluation by the quality of your work is the exception. You have to be particularly good by reputation and metrics in order to reach the stage where your work may be evaluated. And even when your work is evaluated, it's based on superficial reading. The people reading it (who are hopefully experts in the subfield) don't have the time to become experts in that specific niche to judge it properly. What they do is similar to peer review. And like all peer review, it's far more useful for spotting weak points and suggesting improvements than for ranking people/works.