5 ms·

Industry tends to be better because mediocre hires eventually feed into a company's performance against its competitors. Or putting things another way, the publ

Industry tends to be better because mediocre hires eventually feed into a company's performance against its competitors. Or putting things another way, the public sector operates in a fantasy world, whereas the private sector is constantly reality-checked by markets.

nobodyandproud 3y ago
The academic world (nothing to do with the public sector) has its own performance metrics. It also follows it rigidly and in a more cut-throat manner, because there is less jobs. The criticism is that the performance metrics are often in opposition or irrelevant to the stated mission(s) of said academic institutions.
- godelski 3y ago
  > The criticism is that the performance metrics are often in opposition or irrelevant to the stated mission(s) of said academic institutions. This is something I deeply believe. Goals: educate individuals and produce original research. Metrics: h-index (h papers with >= h citations), frequency of publication, academic service (politics). Teaching is generally not considered an important metric. H-index biases to quantity of papers rather than quality, and there is a pseudo-pursuit for novelty while the most important thing in science is reproducibility (hence our crisis). I would say that evaluation of academics is nowhere near aligned with the stated goal of academic institutions and there's a reason several Nobel prize winners have stated that they do not think they would have made it in today's system.
  - Al-Khwarizmi 3y ago
    > Teaching is generally not considered an important metric. The main problem is that no one knows how to measure it. There are many metrics, and as far as I know, all of them are mostly noise. In research, metrics are highly imperfect, but at least they do carry some signal.
    - godelski 3y ago
      > The main problem is that no one knows how to measure it. This is true for every single aspect of academia, even research. I don't think our metrics in research are good and that they are the reason we have the replication crisis. Metrics are always guides, having limitations and biases unique to each one. Over-reliance just results in Goodhart's Law, and instead of embracing the noise inherent to the system we have ill conceived attempts to remove it. Attempts to create meritocratic systems frequently result in less meritocratic systems than were being "improved" upon.
    - nobodyandproud 3y ago
      > The main problem is that no one knows how to measure it. There are many metrics, and as far as I know, all of them are mostly noise. Teaching is a long game. It’s a given that a professor should be knowledgeable in their own topic; but the measure of success for a teacher in higher education is in how knowledgeable and influential their own students are. Contrast Emmy Noether to, say, Isaac Newton. Both were brilliant and loved their topics; but only one was an actual teacher. The other was paranoid and egotistical. Now in matter of influence and metrics: It shouldn’t just be measured in novel research and publishing but the ability to replicate and/or filter out bullshit research as well. Finally, I’d also go as far as to say professionals in industry should be able to link to their old professors and not just institutions. I have so many old, no-name instructors (some dead) who somehow managed to ignite a genuine interest in their topics. I wish I could honor them in a standardized way.
      - godelski 3y ago
        > the measure of success for a teacher in higher education is in how knowledgeable and influential their own students are. this is exactly right, and if thought about carefully, probably impossible to measure without your later suggestion. > only one was an actual teacher. The other was paranoid and egotistical. Hot take. I approve. > shouldn’t just be measured in novel research and publishing but the ability to replicate and/or filter out bullshit research as well. Replication is the foundation of science. It is the only way to actually fight the tyranny of metrics because replication always necessitates variance from the original. Failure to replicate doesn't simply mean fraud, as many believe, but that there are confounding variables that were not uncovered in the original. That's just another piece of the puzzle. But whats destructive is that you can uncover there, formulate a better hypothesis, and then fail to get published due to lack of novelty. Generally associated with reviewers stating that your claim is obvious, despite it not being in the literature or not being able to get anyone to make similar claims prior to giving evidence. There's a reason scissors took a long time to invent, because despite what they look like, it isn't just slapping two knives together. > I’d also go as far as to say professionals in industry should be able to link to their old professors and not just institutions. At least this happens with the PhD level, but I do wonder if this could help at a lower level. Or if this would just end up making the tyranny of metrics worse as people try to extract too strong of a signal? I went to a low rank undergraduate school and I know for a fact that my math education was better than friends who went far more prestigious schools. I took more classes, went in more depth, and was able to solve more Putnam problems than them (only a few total btw, but I did physics not math). With school rankings, I think it is quite silly how we quantify them. I mean if you take schools on csranking.org (at least the top 30) doing a linear regression of the rank against the number of faculty results in a extremely good fit. But I'm not sure why people would be surprised about this given that you're basically limited to teaching at universities ranked lower than the one you graduated from, which should result in homogenization. Yet we still care deeply. Tracking professors sounds like a lot of work, with similarly high variance. I can see utility given that it can help show that you took an easy or a hard professor, but that might just also end up with similar metric hacking where a professor becomes hard rather than successful (because people are likely to measure via failure rate than a more intangible and stochastic long-term outcome based metric which would be coupled with every other educator the student had). It seems easier to just not care. You should get nearly as good outcomes while exerting several orders of magnitude fewer resources. I'm not against it, I just think it would be likely to be far more abused than the current shitty metrics we already use. I think we just at the end of the day need to embrace the stochastic nature of things, and if we don't incorporate that into our models then we're going to end up with bad outcomes. Causal inference is crazy fucking hard because there are far more confounding variables than people assert and more often than not there are elements which are not distinguishable. That many different paths can lead to the same outcome. I'm certainly not saying to give up or abandon metrics, but rather that shits hard and when we start forgetting that we make big mistakes.
godelski 3y ago
Academia isn't supposed to be about markets, and I'd argue that framing it this way is at the root of issues such as the reproducibility crisis. Academia's mission is to educate and research for the public good. These are not things that can be captured well by markets. Success of their goals are fuzzy and take a long time to manifest. This doesn't mean it operates in a fantasy world. Take the difficulty of evaluating employees in industry (which is generally not straight forward), and make it a lot fuzzier but put an even larger stress on these metrics.

nobodyandproud 3y ago

The academic world (nothing to do with the public sector) has its own performance metrics. It also follows it rigidly and in a more cut-throat manner, because there is less jobs. The criticism is that the performance metrics are often in opposition or irrelevant to the stated mission(s) of said academic institutions.

godelski 3y ago
> The criticism is that the performance metrics are often in opposition or irrelevant to the stated mission(s) of said academic institutions. This is something I deeply believe. Goals: educate individuals and produce original research. Metrics: h-index (h papers with >= h citations), frequency of publication, academic service (politics). Teaching is generally not considered an important metric. H-index biases to quantity of papers rather than quality, and there is a pseudo-pursuit for novelty while the most important thing in science is reproducibility (hence our crisis). I would say that evaluation of academics is nowhere near aligned with the stated goal of academic institutions and there's a reason several Nobel prize winners have stated that they do not think they would have made it in today's system.
- Al-Khwarizmi 3y ago
  > Teaching is generally not considered an important metric. The main problem is that no one knows how to measure it. There are many metrics, and as far as I know, all of them are mostly noise. In research, metrics are highly imperfect, but at least they do carry some signal.
  - godelski 3y ago
    > The main problem is that no one knows how to measure it. This is true for every single aspect of academia, even research. I don't think our metrics in research are good and that they are the reason we have the replication crisis. Metrics are always guides, having limitations and biases unique to each one. Over-reliance just results in Goodhart's Law, and instead of embracing the noise inherent to the system we have ill conceived attempts to remove it. Attempts to create meritocratic systems frequently result in less meritocratic systems than were being "improved" upon.
  - nobodyandproud 3y ago
    > The main problem is that no one knows how to measure it. There are many metrics, and as far as I know, all of them are mostly noise. Teaching is a long game. It’s a given that a professor should be knowledgeable in their own topic; but the measure of success for a teacher in higher education is in how knowledgeable and influential their own students are. Contrast Emmy Noether to, say, Isaac Newton. Both were brilliant and loved their topics; but only one was an actual teacher. The other was paranoid and egotistical. Now in matter of influence and metrics: It shouldn’t just be measured in novel research and publishing but the ability to replicate and/or filter out bullshit research as well. Finally, I’d also go as far as to say professionals in industry should be able to link to their old professors and not just institutions. I have so many old, no-name instructors (some dead) who somehow managed to ignite a genuine interest in their topics. I wish I could honor them in a standardized way.
    - godelski 3y ago
      > the measure of success for a teacher in higher education is in how knowledgeable and influential their own students are. this is exactly right, and if thought about carefully, probably impossible to measure without your later suggestion. > only one was an actual teacher. The other was paranoid and egotistical. Hot take. I approve. > shouldn’t just be measured in novel research and publishing but the ability to replicate and/or filter out bullshit research as well. Replication is the foundation of science. It is the only way to actually fight the tyranny of metrics because replication always necessitates variance from the original. Failure to replicate doesn't simply mean fraud, as many believe, but that there are confounding variables that were not uncovered in the original. That's just another piece of the puzzle. But whats destructive is that you can uncover there, formulate a better hypothesis, and then fail to get published due to lack of novelty. Generally associated with reviewers stating that your claim is obvious, despite it not being in the literature or not being able to get anyone to make similar claims prior to giving evidence. There's a reason scissors took a long time to invent, because despite what they look like, it isn't just slapping two knives together. > I’d also go as far as to say professionals in industry should be able to link to their old professors and not just institutions. At least this happens with the PhD level, but I do wonder if this could help at a lower level. Or if this would just end up making the tyranny of metrics worse as people try to extract too strong of a signal? I went to a low rank undergraduate school and I know for a fact that my math education was better than friends who went far more prestigious schools. I took more classes, went in more depth, and was able to solve more Putnam problems than them (only a few total btw, but I did physics not math). With school rankings, I think it is quite silly how we quantify them. I mean if you take schools on csranking.org (at least the top 30) doing a linear regression of the rank against the number of faculty results in a extremely good fit. But I'm not sure why people would be surprised about this given that you're basically limited to teaching at universities ranked lower than the one you graduated from, which should result in homogenization. Yet we still care deeply. Tracking professors sounds like a lot of work, with similarly high variance. I can see utility given that it can help show that you took an easy or a hard professor, but that might just also end up with similar metric hacking where a professor becomes hard rather than successful (because people are likely to measure via failure rate than a more intangible and stochastic long-term outcome based metric which would be coupled with every other educator the student had). It seems easier to just not care. You should get nearly as good outcomes while exerting several orders of magnitude fewer resources. I'm not against it, I just think it would be likely to be far more abused than the current shitty metrics we already use. I think we just at the end of the day need to embrace the stochastic nature of things, and if we don't incorporate that into our models then we're going to end up with bad outcomes. Causal inference is crazy fucking hard because there are far more confounding variables than people assert and more often than not there are elements which are not distinguishable. That many different paths can lead to the same outcome. I'm certainly not saying to give up or abandon metrics, but rather that shits hard and when we start forgetting that we make big mistakes.

godelski 3y ago

Academia isn't supposed to be about markets, and I'd argue that framing it this way is at the root of issues such as the reproducibility crisis. Academia's mission is to educate and research for the public good. These are not things that can be captured well by markets. Success of their goals are fuzzy and take a long time to manifest. This doesn't mean it operates in a fantasy world. Take the difficulty of evaluating employees in industry (which is generally not straight forward), and make it a lot fuzzier but put an even larger stress on these metrics.