5 ms·

It's been said before but once it was decided having papers published became a target everything started to go to shit.

vkou 5y ago
Sure. Which alternative metric would you prefer we use for allocating jobs and funding, and why do you expect that a generation from now, it won't be gamed into absurdity?
jltsiren 5y ago
The root issue is that many people want to become scientists but other people are only willing to fund far fewer scientists. When the supply of scientists far exceeds the demand, there must be a way to decide who gets jobs and funding. Some people favor objective measurements, because they are less prone to corruption and nepotism than subjective ones. But when you have objective measurements, people will try to game them, and some will use dishonest means.
- oconnor663 5y ago
  I think tech company hiring is in a loosely similar situation. No one thinks whiteboard coding is an awesome way to test candidates. Everyone agrees that it doesn't reflect what programmers do day-to-day, and everyone knows there are lots of strong programmers out there who don't do well in whiteboard settings. But I'm not aware of any viable alternatives. The good old fashioned approach of "judge people based on what school they went to, what companies they worked for, and who they know" is probably worse for everyone. The ideal approach might be "hire anyone who asks and then fire them after a year if they're not doing well", but that's very expensive for everyone involved.
  - jimmaswell 5y ago
    There are tons of alternatives, anywhere from take home tests to just gauging their general experience and answers to some technical questions.
    - lisper 5y ago
      Personally I like the idea of hiring a prospective employee as a contractor to do a small but real task. That will give you a more reliable read than any interview. I don’t understand why this is not a more common practice.
      - oconnor663 5y ago
        That does seem like a great approach when it works, but I expect it wouldn't work for the majority of people/positions. A few thoughts off the top of my head: - You need to be available for contracting work, but many people prefer to go from one full-time job to another. Or they might not want to leave their current job, unless they get a much better offer. - You need to be in the right location. Relocating without a job offer is risky. - Especially strong candidates are likely to get many offers when they go looking, which is a good position to be in for salary negotiations. But work-as-interview arrangements make it hard to interview with more than one company at a time. - Anything resembling visa support is probably off the table.
    - Hermitian909 5y ago
      > gauging their general experience and answers to some technical questions This plus a very easy coding question (think fizzbuzz) works really well at a small enough company that you can ensure that all interviewers: 1. Are far enough above the hiring bar that they can comfortably tell when other people are (people near the bar have a harder time distinguishing whether someone is a bit above or below the bar) 2. Are invested enough in the company to want to do a good job interviewing 3. Are thoughtful enough to gauge the competencies the company needs for the role. Those needs are not universal, I know developers who thrive in super technical no-nonsense roles who flounder in product-y work. If a company is very large or growing rapidly it is nearly impossible to guarantee these things. My experience is that companies that don't adopt something like the industry norms lose control of their hiring bar. In some cases this is fine (sometimes you need quantity, not quality) but if the business relied on that high technical bar there's generally a cascading effect of initiatives failing, good engineers leaving, and more initiatives failing because of it. I've seen this tank a few startups in the 50-100 engineer rage.
  - Talanes 5y ago
    One of the standouts of my business school education was the recruiting class where our professor spent most of his time drilling in how bad most of the ways that we assess people are, but that you have to learn to manage to cognitive dissonance of it because that's what most places are going to do anyway.
  - popcube 5y ago
    actually, new professor should publish enough number of research article or they will be fired by organizations
- Emma_Goldman 5y ago
  In my experience in academia, albeit not in the sciences, dishonesty far from the main problem. There is a big difference between fundamental research of wide and lasting interest, and research that ticks enough boxes to be published. Because the second is a lot easier to do than the first, and is what an academic's performance is judged on, everyone ends up spewing out mediocre papers that, intellectually, take us nowhere. Research becomes degraded as a means to the end of a career.
- BeFlatXIII 5y ago
  You see the same dynamic in Hollywood with movie actresses. The biggest difference is that the official grant application racket is replaced with an open-air underground system of sexual favors to studio executives.
bolasanibk 5y ago
When a measure becomes a target, it ceases to be a good measure
- RangerScience 5y ago
  https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law Absolute fav :) Best guesses I've got for how to combat the situation: - Have a diversity of metrics that are somewhat independent, somewhat anti-correlated (if I'm using the right words) - Have non-measurable goals that you refer back to Anyone else got ideas for hedging against the failure mode of Goodhart's Law? :)
  - Maarten88 5y ago
    Yes, changing metrics often in unpredictable ways. I have seen this in a company where they would change employee objectives every year, and also promote and demote people into unexpected new roles. This seemed to result in everyone understanding that next year things can be very different, and does not give people enough time to corrupt a metric. So everyone needs to build good working relations with superiors and subordinates, and not only try to meet objectives but also do well for the company in general, because next year roles can be reversed. This system is tiring for everyone though...
nyolfen 5y ago
you have two options -- empirical metrics or holistic or subjective evaluation. the weakness of the former is that it is subject to gaming, the latter is subject to bias and nepotism.

vkou 5y ago

Sure. Which alternative metric would you prefer we use for allocating jobs and funding, and why do you expect that a generation from now, it won't be gamed into absurdity?

jltsiren 5y ago

The root issue is that many people want to become scientists but other people are only willing to fund far fewer scientists. When the supply of scientists far exceeds the demand, there must be a way to decide who gets jobs and funding. Some people favor objective measurements, because they are less prone to corruption and nepotism than subjective ones. But when you have objective measurements, people will try to game them, and some will use dishonest means.

oconnor663 5y ago
I think tech company hiring is in a loosely similar situation. No one thinks whiteboard coding is an awesome way to test candidates. Everyone agrees that it doesn't reflect what programmers do day-to-day, and everyone knows there are lots of strong programmers out there who don't do well in whiteboard settings. But I'm not aware of any viable alternatives. The good old fashioned approach of "judge people based on what school they went to, what companies they worked for, and who they know" is probably worse for everyone. The ideal approach might be "hire anyone who asks and then fire them after a year if they're not doing well", but that's very expensive for everyone involved.
- jimmaswell 5y ago
  There are tons of alternatives, anywhere from take home tests to just gauging their general experience and answers to some technical questions.
  - lisper 5y ago
    Personally I like the idea of hiring a prospective employee as a contractor to do a small but real task. That will give you a more reliable read than any interview. I don’t understand why this is not a more common practice.
    - oconnor663 5y ago
      That does seem like a great approach when it works, but I expect it wouldn't work for the majority of people/positions. A few thoughts off the top of my head: - You need to be available for contracting work, but many people prefer to go from one full-time job to another. Or they might not want to leave their current job, unless they get a much better offer. - You need to be in the right location. Relocating without a job offer is risky. - Especially strong candidates are likely to get many offers when they go looking, which is a good position to be in for salary negotiations. But work-as-interview arrangements make it hard to interview with more than one company at a time. - Anything resembling visa support is probably off the table.
  - Hermitian909 5y ago
    > gauging their general experience and answers to some technical questions This plus a very easy coding question (think fizzbuzz) works really well at a small enough company that you can ensure that all interviewers: 1. Are far enough above the hiring bar that they can comfortably tell when other people are (people near the bar have a harder time distinguishing whether someone is a bit above or below the bar) 2. Are invested enough in the company to want to do a good job interviewing 3. Are thoughtful enough to gauge the competencies the company needs for the role. Those needs are not universal, I know developers who thrive in super technical no-nonsense roles who flounder in product-y work. If a company is very large or growing rapidly it is nearly impossible to guarantee these things. My experience is that companies that don't adopt something like the industry norms lose control of their hiring bar. In some cases this is fine (sometimes you need quantity, not quality) but if the business relied on that high technical bar there's generally a cascading effect of initiatives failing, good engineers leaving, and more initiatives failing because of it. I've seen this tank a few startups in the 50-100 engineer rage.
- Talanes 5y ago
  One of the standouts of my business school education was the recruiting class where our professor spent most of his time drilling in how bad most of the ways that we assess people are, but that you have to learn to manage to cognitive dissonance of it because that's what most places are going to do anyway.
- popcube 5y ago
  actually, new professor should publish enough number of research article or they will be fired by organizations
Emma_Goldman 5y ago
In my experience in academia, albeit not in the sciences, dishonesty far from the main problem. There is a big difference between fundamental research of wide and lasting interest, and research that ticks enough boxes to be published. Because the second is a lot easier to do than the first, and is what an academic's performance is judged on, everyone ends up spewing out mediocre papers that, intellectually, take us nowhere. Research becomes degraded as a means to the end of a career.
BeFlatXIII 5y ago
You see the same dynamic in Hollywood with movie actresses. The biggest difference is that the official grant application racket is replaced with an open-air underground system of sexual favors to studio executives.

bolasanibk 5y ago

When a measure becomes a target, it ceases to be a good measure

RangerScience 5y ago
https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law Absolute fav :) Best guesses I've got for how to combat the situation: - Have a diversity of metrics that are somewhat independent, somewhat anti-correlated (if I'm using the right words) - Have non-measurable goals that you refer back to Anyone else got ideas for hedging against the failure mode of Goodhart's Law? :)
- Maarten88 5y ago
  Yes, changing metrics often in unpredictable ways. I have seen this in a company where they would change employee objectives every year, and also promote and demote people into unexpected new roles. This seemed to result in everyone understanding that next year things can be very different, and does not give people enough time to corrupt a metric. So everyone needs to build good working relations with superiors and subordinates, and not only try to meet objectives but also do well for the company in general, because next year roles can be reversed. This system is tiring for everyone though...

nyolfen 5y ago

you have two options -- empirical metrics or holistic or subjective evaluation. the weakness of the former is that it is subject to gaming, the latter is subject to bias and nepotism.