7 ms·

I feel like he's kind of invoking Goodhart's law that when a measure becomes a target, it ceases to be a good measure. Academia can only produce what's possible

I feel like he's kind of invoking Goodhart's law that when a measure becomes a target, it ceases to be a good measure. Academia can only produce what's possible to produce under the constraints of meeting certain metrics. If you can't structure the work in such a way as to make the right metrics look good while doing it, you can't do the work at all, because you will be fired, and it doesn't matter how important the work is or why you need to do something that doesn't produce the desired metric to get it done.

theptip 3y ago
Yes, definitely a case of Goodhearting. One way of further breaking this down is the idea of “slicing the bologna” where if you have an idea, you slice it as thin as you can into as many papers as possible. This you can cite your own previous papers, get more totoal citations & papers, and generally game the impact factor. Additionally the “publish or perish” regime encourages small incremental research projects, rather than large expensive risky ones that might not produce a paper for many years. All this against the backdrop that it is getting increasingly harder to find new discoveries since the low-hanging fruit is mostly harvested.
- ivansavz 3y ago
  Lol... "slicing the bologna" is such a great phrase to describe this. I've seen it happen a lot in both experimental and theoretical fields. Rather than publish one big paper that explains a new idea clearly at a high level + show its numerous applications, they publish many (6-10) small results based on the the "machinery" behind this idea. It can be frustrating as a beginner in such fields, like learning from a textbook that shows you just solutions to problems and doesn't explain the general theory.
  - theptip 3y ago
    :) I first heard it in sprinting, where they try to beat the previous world record by the smallest possible margin so they can claim sponsorship bonus from beating it next year too.
atrettel 3y ago
Another way to put this is that the incentives in academia right now encourage "careerism", for lack of a better term. People have to put so much effort into maintaining their position that the actually big picture goals like research become difficult to do, especially when you have to show progress every quarter and funding is only given out in one to two year increments. Researchers end up playing it safe, because big risky ideas may pay off in five to ten years, but that kind of research is hard to fund on a yearly or even quarterly basis.
- opportune 3y ago
  This. The other problem is that as academia becomes a careerist pressure cooker of competition, where the actual science takes a back burner and you’re incentivized to churn out high-likelihood-to-succeed but not-actually-profound science… why even go into academia? Personally, I’d have loved to be a part of the Academia of the mid 20th century. But my exposure to research 10 years ago (thankfully, before committing to a PhD) quickly taught me that it’d be extremely unfulfilling, difficult, and poorly compensated compared to private industry. It would be cool to be a tenured professor I guess, but the coolness and prestige of that title is riding the coattails of the past when academia was less of a slog. I get to work on novel CS problems (distributed systems) in my FAANG job without having to publish X papers per year while I progress on the actual meaty problems, for a lot more pay. It’s just strictly better IMO. I’d rather take 3 years from beginning to end to finish something new I’m actually proud of that’ll be used by hundred of millions of people, than be forced to publish papers very few people will read at a regular cadence or lose my job.
osigurdson 3y ago
I completely reject Goodhart’s “law”. Good metrics are proxies for real things that cannot be gamed. If your metric does not have this quality, delete it.
- tetris11 3y ago
  I like the sentiment, but I'm legitimately scratching my head trying to think of any game-proof metric
  - osigurdson 3y ago
    Salespeople have fairly ungameable metrics - cash. That is the model we should be shooting for. Consumer facing web products value depends clearly on number of users. If a metric cannot be tied to something material, don’t start.
    - nitwit005 3y ago
      It's never that simple. Salespeople frequently want to do things that damage the brand. A lot of companies force them to present pitches a certain way to try to prevent that. And you don't want 50 different sales people all trying to sell to the same client, undermining each other's efforts. That means you need some scheme to assign people to opportunities, such as sales territories. They will learn to game that system.
    - rfrey 3y ago
      Sales are absolutely gameable and regularly are. Sales on false promises to customers, overpromising feature delivery, overstating financial benefits to the customer, are so common as to be cliche. The customers end up being short term relationships, the sales provide negative value to the company due to reputation loss (not realized for perhaps years), but the salesperson collects the commission and is probably working somewhere else when the smoke clears. And I'm not sure if you're claiming "number of users" is ungameable. If so, I'd invite you to reconsider that - we've just lived through an entire era where companies gamed user metrics to harvest sweet, sweet VC dollars. Founders game the metrics for VCs, vice presidents game the metrics for founders, product managers game the metrics for VPs.

theptip 3y ago

Yes, definitely a case of Goodhearting. One way of further breaking this down is the idea of “slicing the bologna” where if you have an idea, you slice it as thin as you can into as many papers as possible. This you can cite your own previous papers, get more totoal citations & papers, and generally game the impact factor. Additionally the “publish or perish” regime encourages small incremental research projects, rather than large expensive risky ones that might not produce a paper for many years. All this against the backdrop that it is getting increasingly harder to find new discoveries since the low-hanging fruit is mostly harvested.

ivansavz 3y ago
Lol... "slicing the bologna" is such a great phrase to describe this. I've seen it happen a lot in both experimental and theoretical fields. Rather than publish one big paper that explains a new idea clearly at a high level + show its numerous applications, they publish many (6-10) small results based on the the "machinery" behind this idea. It can be frustrating as a beginner in such fields, like learning from a textbook that shows you just solutions to problems and doesn't explain the general theory.
- theptip 3y ago
  :) I first heard it in sprinting, where they try to beat the previous world record by the smallest possible margin so they can claim sponsorship bonus from beating it next year too.

atrettel 3y ago

Another way to put this is that the incentives in academia right now encourage "careerism", for lack of a better term. People have to put so much effort into maintaining their position that the actually big picture goals like research become difficult to do, especially when you have to show progress every quarter and funding is only given out in one to two year increments. Researchers end up playing it safe, because big risky ideas may pay off in five to ten years, but that kind of research is hard to fund on a yearly or even quarterly basis.

opportune 3y ago
This. The other problem is that as academia becomes a careerist pressure cooker of competition, where the actual science takes a back burner and you’re incentivized to churn out high-likelihood-to-succeed but not-actually-profound science… why even go into academia? Personally, I’d have loved to be a part of the Academia of the mid 20th century. But my exposure to research 10 years ago (thankfully, before committing to a PhD) quickly taught me that it’d be extremely unfulfilling, difficult, and poorly compensated compared to private industry. It would be cool to be a tenured professor I guess, but the coolness and prestige of that title is riding the coattails of the past when academia was less of a slog. I get to work on novel CS problems (distributed systems) in my FAANG job without having to publish X papers per year while I progress on the actual meaty problems, for a lot more pay. It’s just strictly better IMO. I’d rather take 3 years from beginning to end to finish something new I’m actually proud of that’ll be used by hundred of millions of people, than be forced to publish papers very few people will read at a regular cadence or lose my job.

osigurdson 3y ago

I completely reject Goodhart’s “law”. Good metrics are proxies for real things that cannot be gamed. If your metric does not have this quality, delete it.

tetris11 3y ago
I like the sentiment, but I'm legitimately scratching my head trying to think of any game-proof metric
- osigurdson 3y ago
  Salespeople have fairly ungameable metrics - cash. That is the model we should be shooting for. Consumer facing web products value depends clearly on number of users. If a metric cannot be tied to something material, don’t start.
  - nitwit005 3y ago
    It's never that simple. Salespeople frequently want to do things that damage the brand. A lot of companies force them to present pitches a certain way to try to prevent that. And you don't want 50 different sales people all trying to sell to the same client, undermining each other's efforts. That means you need some scheme to assign people to opportunities, such as sales territories. They will learn to game that system.
  - rfrey 3y ago
    Sales are absolutely gameable and regularly are. Sales on false promises to customers, overpromising feature delivery, overstating financial benefits to the customer, are so common as to be cliche. The customers end up being short term relationships, the sales provide negative value to the company due to reputation loss (not realized for perhaps years), but the salesperson collects the commission and is probably working somewhere else when the smoke clears. And I'm not sure if you're claiming "number of users" is ungameable. If so, I'd invite you to reconsider that - we've just lived through an entire era where companies gamed user metrics to harvest sweet, sweet VC dollars. Founders game the metrics for VCs, vice presidents game the metrics for founders, product managers game the metrics for VPs.