5 ms·

Two ways I interpret this. 1. Academia has fallen into a self consuming game where fierce individual competition to appear productive means that researchers do

Two ways I interpret this. 1. Academia has fallen into a self consuming game where fierce individual competition to appear productive means that researchers don't have the space to actually think creatively and come up with incredible new insights. 2. Scientists like Higgs had the privilege of lower hanging fruit to harvest in their day, so a career of lesser intensity was sufficient for people in his cohort to produce incredible contributions. Modern scientists legitimately have to work harder to produce less profound results. Maybe there's something to both lines of thinking. I'm also reminded of the classic story of the pottery teacher who graded one class by final project quality and another by total weight produced over the class period, and found that the latter group ended up producing higher quality work. Higgs in this article seems to be of the persuasion that in his profession, the former style produces better results. I think many of us here would find that intuitively wrong in general.

theonemind 3y ago
I feel like he's kind of invoking Goodhart's law that when a measure becomes a target, it ceases to be a good measure. Academia can only produce what's possible to produce under the constraints of meeting certain metrics. If you can't structure the work in such a way as to make the right metrics look good while doing it, you can't do the work at all, because you will be fired, and it doesn't matter how important the work is or why you need to do something that doesn't produce the desired metric to get it done.
- theptip 3y ago
  Yes, definitely a case of Goodhearting. One way of further breaking this down is the idea of “slicing the bologna” where if you have an idea, you slice it as thin as you can into as many papers as possible. This you can cite your own previous papers, get more totoal citations & papers, and generally game the impact factor. Additionally the “publish or perish” regime encourages small incremental research projects, rather than large expensive risky ones that might not produce a paper for many years. All this against the backdrop that it is getting increasingly harder to find new discoveries since the low-hanging fruit is mostly harvested.
  - ivansavz 3y ago
    Lol... "slicing the bologna" is such a great phrase to describe this. I've seen it happen a lot in both experimental and theoretical fields. Rather than publish one big paper that explains a new idea clearly at a high level + show its numerous applications, they publish many (6-10) small results based on the the "machinery" behind this idea. It can be frustrating as a beginner in such fields, like learning from a textbook that shows you just solutions to problems and doesn't explain the general theory.
    - theptip 3y ago
      :) I first heard it in sprinting, where they try to beat the previous world record by the smallest possible margin so they can claim sponsorship bonus from beating it next year too.
- atrettel 3y ago
  Another way to put this is that the incentives in academia right now encourage "careerism", for lack of a better term. People have to put so much effort into maintaining their position that the actually big picture goals like research become difficult to do, especially when you have to show progress every quarter and funding is only given out in one to two year increments. Researchers end up playing it safe, because big risky ideas may pay off in five to ten years, but that kind of research is hard to fund on a yearly or even quarterly basis.
  - opportune 3y ago
    This. The other problem is that as academia becomes a careerist pressure cooker of competition, where the actual science takes a back burner and you’re incentivized to churn out high-likelihood-to-succeed but not-actually-profound science… why even go into academia? Personally, I’d have loved to be a part of the Academia of the mid 20th century. But my exposure to research 10 years ago (thankfully, before committing to a PhD) quickly taught me that it’d be extremely unfulfilling, difficult, and poorly compensated compared to private industry. It would be cool to be a tenured professor I guess, but the coolness and prestige of that title is riding the coattails of the past when academia was less of a slog. I get to work on novel CS problems (distributed systems) in my FAANG job without having to publish X papers per year while I progress on the actual meaty problems, for a lot more pay. It’s just strictly better IMO. I’d rather take 3 years from beginning to end to finish something new I’m actually proud of that’ll be used by hundred of millions of people, than be forced to publish papers very few people will read at a regular cadence or lose my job.
- osigurdson 3y ago
  I completely reject Goodhart’s “law”. Good metrics are proxies for real things that cannot be gamed. If your metric does not have this quality, delete it.
  - tetris11 3y ago
    I like the sentiment, but I'm legitimately scratching my head trying to think of any game-proof metric
    - osigurdson 3y ago
      Salespeople have fairly ungameable metrics - cash. That is the model we should be shooting for. Consumer facing web products value depends clearly on number of users. If a metric cannot be tied to something material, don’t start.
      - nitwit005 3y ago
        It's never that simple. Salespeople frequently want to do things that damage the brand. A lot of companies force them to present pitches a certain way to try to prevent that. And you don't want 50 different sales people all trying to sell to the same client, undermining each other's efforts. That means you need some scheme to assign people to opportunities, such as sales territories. They will learn to game that system.
      - rfrey 3y ago
        Sales are absolutely gameable and regularly are. Sales on false promises to customers, overpromising feature delivery, overstating financial benefits to the customer, are so common as to be cliche. The customers end up being short term relationships, the sales provide negative value to the company due to reputation loss (not realized for perhaps years), but the salesperson collects the commission and is probably working somewhere else when the smoke clears. And I'm not sure if you're claiming "number of users" is ungameable. If so, I'd invite you to reconsider that - we've just lived through an entire era where companies gamed user metrics to harvest sweet, sweet VC dollars. Founders game the metrics for VCs, vice presidents game the metrics for founders, product managers game the metrics for VPs.
contravariant 3y ago
If the low hanging fruit is gone and producing profound results requires more effort you'd expect the number of papers to decrease, or at least it wouldn't be appropriate to use number of papers as a metric because producing good papers requires exceedingly large amounts of efforts and possibly luck.
- thaw13579 3y ago
  While that seems reasonable, it assumes the amount of impact per paper has remained constant, while is practice it has greatly declined. These days scientists will literally try to identify the set of "least publishable units" in a research project to maximize the number of papers (I've attended meetings where this is discussed without irony). There is also a tendency to publish work that is quite similar to previous work with a minor change that is emphasized disproportionately to its importance.
- noslenwerdna 3y ago
  Not if the scientists involved are p-hacking. Or, less harmfully, publishing null results.
  - iraqmtpizza 3y ago
    Publishing null results is the opposite of harmful
  - piperswe 3y ago
    Surely you mean _not_ publishing null results? From what I understand, null results are just as scientifically important to publish as alternative results, they just aren't published as often because they aren't as exciting.
- solveit 3y ago
  Yes, I would expect amount of work per paper to stay roughly constant in a healthy research community. The whole point of writing papers should be to communicate what you did and found so that others don't have to do the same thing, and I think the amount of work would be the relevant variable that determines the amount of information that has to be transferred.
imbnwa 3y ago
>I'm also reminded of the classic story of the pottery teacher who graded one class by final project quality and another by total weight produced over the class period, and found that the latter group ended up producing higher quality work. Higgs in this article seems to be of the persuasion that in his profession, the former style produces better results. I think many of us here would find that intuitively wrong in general. Wait, how is pottery equivalent to significant expansion of the bounds of scientific knowledge? Humans mastered the contours of pottery thousands of years before farming was a thing, we have no such bearing wrt scientific knowledge.
hibikir 3y ago
Doing science and publishing papers is extremely different than pottery, and that has nothing to do with a difference in status between the two. When doing any form of art, the most important thing is to practice. If you make 20 vases, the 20th is most likely far better than the 1st. Same with painting, drawing, other forms of sculpture... nothing teaches like practice. There's such thing as more complicated pieces that require more work, but the majority of actual skill at pottery is practiced the same way whether you are aiming for a relatively complex piece, or one that is just hard enough it doesn't bore you. On the other hand, in science, there's no such thing as random practice making you better. The more papers you publish, the higher percentage of the time is dedicated to the writing bits, and getting through the approvals, and doing a million submissions, instead of coming up with good hypothesis, and finding something to do. Aiming at a hard problem is going to lead you to publishing less, not more, so the things you'll work on are going to be minor improvements, nothing difficult. Therefore, the chances that you'll discover anything major definitely decrease the more often you publish. So if in your intuition you'd be better off focusing on making a lot of unimportant papers... why would anyone aim for any fundamental problem, which has a higher chance of failure? You think the efforts to actually publish and submit, all away from the lab aren't mostly wasteful? Because it's really hard for me to imagine how publishing more doesn't mean far less time doing research.
- ansgri 3y ago
  Interesting analogy with pottery. I see that an important academic practice that does scale like vases is structuring your research in easily publishable way. As a beginner (typically a young highly motivated idealistic student — who else would want to go to academia in the first place?), you can start with good research idea and important results and no understanding of academic world, but struggle to get accepted anywhere decent because you didn't follow all unwritten rules, and as you get experience, you learn the rules of the game and plan your work to contain many low-risk "least publishable units" to build your reputation. So yes, "a lot of unimportant papers". Many interesting papers nowadays come from commercial R&D departments, like MS Research, because unimportant papers probably don't move your main KPIs (which would be "a lot of patents" AFAIU), and only interesting results improve the image of the company and serve as good marketing material.

theonemind 3y ago

I feel like he's kind of invoking Goodhart's law that when a measure becomes a target, it ceases to be a good measure. Academia can only produce what's possible to produce under the constraints of meeting certain metrics. If you can't structure the work in such a way as to make the right metrics look good while doing it, you can't do the work at all, because you will be fired, and it doesn't matter how important the work is or why you need to do something that doesn't produce the desired metric to get it done.

theptip 3y ago
Yes, definitely a case of Goodhearting. One way of further breaking this down is the idea of “slicing the bologna” where if you have an idea, you slice it as thin as you can into as many papers as possible. This you can cite your own previous papers, get more totoal citations & papers, and generally game the impact factor. Additionally the “publish or perish” regime encourages small incremental research projects, rather than large expensive risky ones that might not produce a paper for many years. All this against the backdrop that it is getting increasingly harder to find new discoveries since the low-hanging fruit is mostly harvested.
- ivansavz 3y ago
  Lol... "slicing the bologna" is such a great phrase to describe this. I've seen it happen a lot in both experimental and theoretical fields. Rather than publish one big paper that explains a new idea clearly at a high level + show its numerous applications, they publish many (6-10) small results based on the the "machinery" behind this idea. It can be frustrating as a beginner in such fields, like learning from a textbook that shows you just solutions to problems and doesn't explain the general theory.
  - theptip 3y ago
    :) I first heard it in sprinting, where they try to beat the previous world record by the smallest possible margin so they can claim sponsorship bonus from beating it next year too.
atrettel 3y ago
Another way to put this is that the incentives in academia right now encourage "careerism", for lack of a better term. People have to put so much effort into maintaining their position that the actually big picture goals like research become difficult to do, especially when you have to show progress every quarter and funding is only given out in one to two year increments. Researchers end up playing it safe, because big risky ideas may pay off in five to ten years, but that kind of research is hard to fund on a yearly or even quarterly basis.
- opportune 3y ago
  This. The other problem is that as academia becomes a careerist pressure cooker of competition, where the actual science takes a back burner and you’re incentivized to churn out high-likelihood-to-succeed but not-actually-profound science… why even go into academia? Personally, I’d have loved to be a part of the Academia of the mid 20th century. But my exposure to research 10 years ago (thankfully, before committing to a PhD) quickly taught me that it’d be extremely unfulfilling, difficult, and poorly compensated compared to private industry. It would be cool to be a tenured professor I guess, but the coolness and prestige of that title is riding the coattails of the past when academia was less of a slog. I get to work on novel CS problems (distributed systems) in my FAANG job without having to publish X papers per year while I progress on the actual meaty problems, for a lot more pay. It’s just strictly better IMO. I’d rather take 3 years from beginning to end to finish something new I’m actually proud of that’ll be used by hundred of millions of people, than be forced to publish papers very few people will read at a regular cadence or lose my job.
osigurdson 3y ago
I completely reject Goodhart’s “law”. Good metrics are proxies for real things that cannot be gamed. If your metric does not have this quality, delete it.
- tetris11 3y ago
  I like the sentiment, but I'm legitimately scratching my head trying to think of any game-proof metric
  - osigurdson 3y ago
    Salespeople have fairly ungameable metrics - cash. That is the model we should be shooting for. Consumer facing web products value depends clearly on number of users. If a metric cannot be tied to something material, don’t start.
    - nitwit005 3y ago
      It's never that simple. Salespeople frequently want to do things that damage the brand. A lot of companies force them to present pitches a certain way to try to prevent that. And you don't want 50 different sales people all trying to sell to the same client, undermining each other's efforts. That means you need some scheme to assign people to opportunities, such as sales territories. They will learn to game that system.
    - rfrey 3y ago
      Sales are absolutely gameable and regularly are. Sales on false promises to customers, overpromising feature delivery, overstating financial benefits to the customer, are so common as to be cliche. The customers end up being short term relationships, the sales provide negative value to the company due to reputation loss (not realized for perhaps years), but the salesperson collects the commission and is probably working somewhere else when the smoke clears. And I'm not sure if you're claiming "number of users" is ungameable. If so, I'd invite you to reconsider that - we've just lived through an entire era where companies gamed user metrics to harvest sweet, sweet VC dollars. Founders game the metrics for VCs, vice presidents game the metrics for founders, product managers game the metrics for VPs.

contravariant 3y ago

If the low hanging fruit is gone and producing profound results requires more effort you'd expect the number of papers to decrease, or at least it wouldn't be appropriate to use number of papers as a metric because producing good papers requires exceedingly large amounts of efforts and possibly luck.

thaw13579 3y ago
While that seems reasonable, it assumes the amount of impact per paper has remained constant, while is practice it has greatly declined. These days scientists will literally try to identify the set of "least publishable units" in a research project to maximize the number of papers (I've attended meetings where this is discussed without irony). There is also a tendency to publish work that is quite similar to previous work with a minor change that is emphasized disproportionately to its importance.
noslenwerdna 3y ago
Not if the scientists involved are p-hacking. Or, less harmfully, publishing null results.
- iraqmtpizza 3y ago
  Publishing null results is the opposite of harmful
- piperswe 3y ago
  Surely you mean _not_ publishing null results? From what I understand, null results are just as scientifically important to publish as alternative results, they just aren't published as often because they aren't as exciting.
solveit 3y ago
Yes, I would expect amount of work per paper to stay roughly constant in a healthy research community. The whole point of writing papers should be to communicate what you did and found so that others don't have to do the same thing, and I think the amount of work would be the relevant variable that determines the amount of information that has to be transferred.

imbnwa 3y ago

>I'm also reminded of the classic story of the pottery teacher who graded one class by final project quality and another by total weight produced over the class period, and found that the latter group ended up producing higher quality work. Higgs in this article seems to be of the persuasion that in his profession, the former style produces better results. I think many of us here would find that intuitively wrong in general. Wait, how is pottery equivalent to significant expansion of the bounds of scientific knowledge? Humans mastered the contours of pottery thousands of years before farming was a thing, we have no such bearing wrt scientific knowledge.

hibikir 3y ago

Doing science and publishing papers is extremely different than pottery, and that has nothing to do with a difference in status between the two. When doing any form of art, the most important thing is to practice. If you make 20 vases, the 20th is most likely far better than the 1st. Same with painting, drawing, other forms of sculpture... nothing teaches like practice. There's such thing as more complicated pieces that require more work, but the majority of actual skill at pottery is practiced the same way whether you are aiming for a relatively complex piece, or one that is just hard enough it doesn't bore you. On the other hand, in science, there's no such thing as random practice making you better. The more papers you publish, the higher percentage of the time is dedicated to the writing bits, and getting through the approvals, and doing a million submissions, instead of coming up with good hypothesis, and finding something to do. Aiming at a hard problem is going to lead you to publishing less, not more, so the things you'll work on are going to be minor improvements, nothing difficult. Therefore, the chances that you'll discover anything major definitely decrease the more often you publish. So if in your intuition you'd be better off focusing on making a lot of unimportant papers... why would anyone aim for any fundamental problem, which has a higher chance of failure? You think the efforts to actually publish and submit, all away from the lab aren't mostly wasteful? Because it's really hard for me to imagine how publishing more doesn't mean far less time doing research.

ansgri 3y ago
Interesting analogy with pottery. I see that an important academic practice that does scale like vases is structuring your research in easily publishable way. As a beginner (typically a young highly motivated idealistic student — who else would want to go to academia in the first place?), you can start with good research idea and important results and no understanding of academic world, but struggle to get accepted anywhere decent because you didn't follow all unwritten rules, and as you get experience, you learn the rules of the game and plan your work to contain many low-risk "least publishable units" to build your reputation. So yes, "a lot of unimportant papers". Many interesting papers nowadays come from commercial R&D departments, like MS Research, because unimportant papers probably don't move your main KPIs (which would be "a lot of patents" AFAIU), and only interesting results improve the image of the company and serve as good marketing material.