11 ms·

My theory is that as more people compete, the top candidates become those who are best at gaming the system rather than actually being the best. Someone has pro

My theory is that as more people compete, the top candidates become those who are best at gaming the system rather than actually being the best. Someone has probably studied this. My only evidence is job applications for GAFAM and Tinder tho.

godelski 1y ago
> Someone has probably studied this There's even a name for it https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
- julienreszka 1y ago
  It’s a false law tho. Collapses under scrutiny
  - godelski 1y ago
    Sorry, remind me; how many cobras are there in India?
    - bandrami 1y ago
      The Zoological Survey of India would like to know but hasn't figured out a good way to do a full census. If you have any ideas they would love to hear them. Naja naja has Least Concern conservation status, so there isn't much funding in doing a full count, but there are concerns as encroachment both reduces their livable habitat and puts them into more frequent contact with humans and livestock.
      - oblio 1y ago
        The comment was a joke. https://en.wikipedia.org/wiki/Perverse_incentive https://en.wikipedia.org/wiki/Perverse_incentive
  - epwr 1y ago
    Could you elaborate or link something here? I think about this pretty frequently, so would love to read something!
    - vasco 1y ago
      Metric: time to run 100m Context: track athlete Does it cease to be a good metric? No. After this you can likely come up with many examples of target metrics which never turn bad.
      - ccortes 1y ago
        > Does it cease to be a good metric? Yes if you run anything other than the 100m
      - godelski 1y ago
        So what is your argument, that it doesn't apply everywhere therefore it applies nowhere? You're misunderstanding the root cause. Your example works as the the metric is well aligned. I'm sure you can also think of many examples where the metric is not well aligned and maximizing it becomes harmful. How do you think we ended up with clickbait titles? Why was everyone so focused on clicks? Let's think about engagement metrics. Is that what we really want to measure? Do we have no preference over users being happy vs users being angry or sad? Or are those things much harder to measure, if not impossible to, and thus we focus on our proxies instead? So what happens when someone doesn't realize it is a proxy and becomes hyper fixated on it? What happens if someone does realize it is a proxy but is rewarded via the metric so they don't really care? Your example works in the simple case, but a lot of things look trivial when you only approach them from a first order approximation. You left out all the hard stuff. It's kinda like... Edit: Looks like some people are bringing up metric limits that I couldn't come up with. Thanks!
        vasco 1y ago
        > So what is your argument, that it doesn't apply everywhere therefore it applies nowhere? I never said that. Someone said the law collapses, someone asked for a link, I gave an example to prove it does break down in some cases at least, but many cases once you think more about it. I never said all cases. If it works sometimes and not others, it's not a law. It's just an observation of something that can happen or not.
  - NBJack 1y ago
    If I hadn't seen it in action countless times, I would belive you. Changelists, line counts, documents made, collaborator counts, teams lead, reference counts in peer reviewed journals...the list goes on. You are welcome to prove me wrong though. You might even restore some faith in humanity, too!
- ivanbelenky 1y ago
  Thanks for sharing. I did not know this law existed and had a name. I know nothing about nothing but it appears to be the case that the interpretation of metrics for policies assume implicitly the "shape" of the domain. E.g. in RL for games we see a bunch of outlier behavior for policies just gaming the signal. There seems to be 2 types - Specification failure: signal is bad-ish, a completely broken behavior --> local optimal points achieved for policies that phenomenologically do not represent what was expected/desired to cover --> signaling an improvable reward signal definition - Domain constraint failure: signal is still good and optimization is "legitimate", but you are prompted with the question "do I need to constraint my domain of solutions?" - finding a bug that reduces time to completion of a game in a speedrun setting would be a new acceptable baseline, because there are no rules to finishing the game earlier - shooting amphetamines on a 100m run would probably minimize time, but other factors will make people consider disallowing such practices.
  - Eisenstein 1y ago
    I view Goodhart's law more as a lesson for why we can never achieve a goal by offering specific incentives if we are measuring success by the outcome of the incentives and not by the achievement of the goal. This is of course inevitable if the goal cannot be directly measured but is composed of many constantly moving variables such as education or public health. This doesn't mean we shouldn't bother having such goals, it just means we have to be diligent at pivoting the incentives when it becomes evident that secondary effects are being produced at the expense of the desired effect.
    - godelski 1y ago
      > This is of course inevitable if the goal cannot be directly measured It's worth noting that no goal can be directly measured[0]. I agree with you, this doesn't mean we shouldn't bother with goals. They are fantastic tools. But they are guides. The better aligned our proxy measurement is with the intended measurement then the less we have to interpret our results. We have to think less, spending less energy. But even poorly defined goals can be helpful, as they get refined as we progress in them. We've all done this since we were kids and we do this to this day. All long term goals are updated as we progress in them. It's not like we just state a goal and then hop on the railroad to success. It's like writing tests for code. Tests don't prove that your code is bug free (can't write a test for a bug you don't know about: unknown unknown). But tests are still helpful because they help evidence the code is bug free and constrain the domain in which bugs can live. It's also why TDD is naive, because tests aren't proof and you have to continue to think beyond the tests. [0] https://news.ycombinator.com/item?id=45555551 https://news.ycombinator.com/item?id=45555551
bjornsing 1y ago
Yeah I think this is a general principle. Just look at the quality of US presidents over time, or generations of top physicists. I guess it’s just a numbers game: the number of genuinely interested people is relatively constant while the number of gamers grows with the compensation and perceived status of the activity. So when compensation and perceived status skyrockets the ratio between those numbers changes drastically.
- godelski 1y ago
  I think the number of generally interested people goes up. Maybe the percent stays the same? But honestly, I think we kill passion for a lot of people. To be cliche, how many people lose the curiosity of a child? I think the cliche exists for a reason. It seems the capacity is in all of us and even once existed.
  - bjornsing 1y ago
    To some extent I think that’s just human nature, or even animal nature. The optimal explore / exploit tradeoff changes as we age. When we’re children it’s beneficial to explore. As adults it’s often more beneficial to exploit. But you need cultural and organizational safeguards that protect those of us who are more childish and explorative from those that are more cynical and exploitative. Otherwise pursuits of truth aren’t very fruitful.
crystal_revenge 1y ago
I've spent most of my career working, chatting and hanging out with what might be best described as "passionate weirdos" in various quantitative areas of research. I say "weirdos" because they're people driven by an obsession with a topic, but don't always fit the mold by having the ideal combination of background, credentials and personality to land them on a big tech company research team. The other day I was spending some time with a researcher from Deep Mind and I was surprised to find that while they were sharp and curious to an extent, nearly every ounce of energy they expended on research was strategic. They didn't write about research they were fascinated by, they wrote and researched on topics they strategically felt had the highest probability getting into a major conference in a short period of time to earn them a promotion. While I was a bit disappointed, I certainly didn't judge them because they are just playing the game. This person probably earns more than many rooms of smart, passionate people I've been in, and that money isn't for smarts alone; it's for appealing to the interests of people with the money. You can see this very clearly by comparing the work being done in the LLM space to that being done in the Image/Video diffusion model space. There's much more money in LLMs right now, and the field is flooded with papers on any random topic. If you dive in, most of them are not reproducible or make very questionable conclusions based on the data they present, but that's not of very much concern so long as the paper can be added to a CV. In the stable diffusion world it's mostly people driven by personal interest (usually very non-commericial personal interests) and you see tons of innovation in that field but almost no papers. In fact, if you really want to understand a lot of the most novel work coming out of the image generation world you often need to dig into PRs made by an anonymous users with anime themed profile pic. The bummer of course is that there are very hard limits on what any researcher can do with a home GPU training setup. It does lead to creative solutions to problems, but I can't help but wonder what the world would look like if more of these people had even a fraction of the resources available exclusively to people playing the game.
- smokel 1y ago
  > I certainly didn't judge them because they are just playing the game. Please do judge them for being parasitical. They might seem successful by certain measures, like the amount of money they make, but I for one simply dislike it when people only think about themselves. As a society, we should be more cautious about narcissism and similar behaviors. Also, in the long run, this kind of behaviour makes them an annoying person at parties.
xvector 1y ago
I have seen absolutely incredible, best in the world type engineers, much smarter than myself, get fired from my FAANG because of the performance games. I persist because I'm fantastic at politics while being good enough to do my job. Feels weird man.
t_serpico 1y ago
But there is no way to know who is truly the 'best'. The people who position and market themselves to be viewed as the best are the only ones who even have a chance to be viewed as such. So if you're a great researcher but don't project yourself that way, no one will ever know you're a great researcher (except for the other great researchers who aren't really invested in communicating how great you are). The system seems to incentivize people to not only optimize for their output but also their image. This isn't a bad thing per se, but is sort of antithetical to the whole shoulder of giants ethos of science.
- kcexn 1y ago
  The problem is that the best research is not a competitive process but a collaborative one. Positioning research output as a race or a competition is already problematic.
  - bwfan123 1y ago
    right. Also, the idea that there is a "best" researcher is already problematic. You could have 10 great people in a team, and it would be hard to rank them. Rating people in order of performance in a team is contradictory to the idea of building a great team. ie, you could have 10 people all rated 10 which is really the goal when building a team.
rightbyte 1y ago
This is an interesting theory. I think there is something to it. It is really hard to do good in a competitive environment. Very constrained.
meindnoch 1y ago
Goodhart's law
RataNova 1y ago
Anytime a system gets hyper-competitive and the stakes are high, it starts selecting for people who are good at playing the system rather than just excelling at the underlying skill
bwfan123 1y ago
I would categorize people into 2 broad extremes. 1) those that care two hoots about what others or the system expects of them and in that sense are authentic and 2) those that only care about what others or the system expects of them, and in that sense are not authentic. There is a spectrum in there.
- godelski 1y ago
  If you haven't heard this already, you might be interested in Pournelle's Iron Law of Bureaucracy https://jerrypournelle.com/reports/jerryp/iron.html https://jerrypournelle.com/reports/jerryp/iron.html
b00ty4breakfast 1y ago
that's what happens at the top of most competitive domains. Just take a look at pro sports; guys are looking for millimeters to shave off and they turn to "playing the game" rather than merely improving athletic performance. Watching a football game (either kind) and a not-small portion of the action is guys trying to draw penalties or exploit the rules to get an edge.
nathan_compton 1y ago
It is pretty simple - if the rewards are great enough and the objective difficult enough, at some point it becomes more efficient to kneecap your competitors rather than to try to outrun them. I genuinely thing science would be better served if scientist got paid modest salaries to pursue their own research interests and all results became public domain. So many Universities now fancy themselves startup factories, and startups are great for some things, no doubt, but I don't think pure research is always served by this strategy.
- godelski 1y ago
  > if scientist got paid modest salaries to pursue their own research interests and all results became public domain I would make that deal in a heartbeat[0,1]. We made a mistake by making academia a business. The point was that certain research creates the foundation for others to stand on, but it is difficult to profit off those innovations and by making those innovations public then the society at large will profit by several orders of magnitude more than you would have if you could have. Newton and Leibniz didn't become billionaires by inventing calculus, yet we wouldn't have the trillion dollar businesses and half the technology we have today if they hadn't. You could say the same about Tim Burner Lee's innovation. The idea that we have to justify our research and sell it as profitable is insane. It is as if being unaware of the past itself. Yeah, there's lots of failures in research, it's hard to push the bounds of human knowledge (surprise?). But there are hundreds, if not millions, of examples where that innovation results in so much value that the entire global revenue is not enough. Because the entire global revenue stands on this very foundation. I'm not saying scientists need to be billionaires, but it's fucking ridiculous that we have to fight so hard to justify buying a fucking laptop. It is beyond absurd. [0] https://news.ycombinator.com/item?id=45422828 https://news.ycombinator.com/item?id=45422828 [1] https://news.ycombinator.com/item?id=43959309 https://news.ycombinator.com/item?id=43959309

godelski 1y ago

> Someone has probably studied this There's even a name for it https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law

julienreszka 1y ago
It’s a false law tho. Collapses under scrutiny
- godelski 1y ago
  Sorry, remind me; how many cobras are there in India?
  - bandrami 1y ago
    The Zoological Survey of India would like to know but hasn't figured out a good way to do a full census. If you have any ideas they would love to hear them. Naja naja has Least Concern conservation status, so there isn't much funding in doing a full count, but there are concerns as encroachment both reduces their livable habitat and puts them into more frequent contact with humans and livestock.
    - oblio 1y ago
      The comment was a joke. https://en.wikipedia.org/wiki/Perverse_incentive https://en.wikipedia.org/wiki/Perverse_incentive
- epwr 1y ago
  Could you elaborate or link something here? I think about this pretty frequently, so would love to read something!
  - vasco 1y ago
    Metric: time to run 100m Context: track athlete Does it cease to be a good metric? No. After this you can likely come up with many examples of target metrics which never turn bad.
    - ccortes 1y ago
      > Does it cease to be a good metric? Yes if you run anything other than the 100m
    - godelski 1y ago
      So what is your argument, that it doesn't apply everywhere therefore it applies nowhere? You're misunderstanding the root cause. Your example works as the the metric is well aligned. I'm sure you can also think of many examples where the metric is not well aligned and maximizing it becomes harmful. How do you think we ended up with clickbait titles? Why was everyone so focused on clicks? Let's think about engagement metrics. Is that what we really want to measure? Do we have no preference over users being happy vs users being angry or sad? Or are those things much harder to measure, if not impossible to, and thus we focus on our proxies instead? So what happens when someone doesn't realize it is a proxy and becomes hyper fixated on it? What happens if someone does realize it is a proxy but is rewarded via the metric so they don't really care? Your example works in the simple case, but a lot of things look trivial when you only approach them from a first order approximation. You left out all the hard stuff. It's kinda like... Edit: Looks like some people are bringing up metric limits that I couldn't come up with. Thanks!
      - vasco 1y ago
        > So what is your argument, that it doesn't apply everywhere therefore it applies nowhere? I never said that. Someone said the law collapses, someone asked for a link, I gave an example to prove it does break down in some cases at least, but many cases once you think more about it. I never said all cases. If it works sometimes and not others, it's not a law. It's just an observation of something that can happen or not.
- NBJack 1y ago
  If I hadn't seen it in action countless times, I would belive you. Changelists, line counts, documents made, collaborator counts, teams lead, reference counts in peer reviewed journals...the list goes on. You are welcome to prove me wrong though. You might even restore some faith in humanity, too!
ivanbelenky 1y ago
Thanks for sharing. I did not know this law existed and had a name. I know nothing about nothing but it appears to be the case that the interpretation of metrics for policies assume implicitly the "shape" of the domain. E.g. in RL for games we see a bunch of outlier behavior for policies just gaming the signal. There seems to be 2 types - Specification failure: signal is bad-ish, a completely broken behavior --> local optimal points achieved for policies that phenomenologically do not represent what was expected/desired to cover --> signaling an improvable reward signal definition - Domain constraint failure: signal is still good and optimization is "legitimate", but you are prompted with the question "do I need to constraint my domain of solutions?" - finding a bug that reduces time to completion of a game in a speedrun setting would be a new acceptable baseline, because there are no rules to finishing the game earlier - shooting amphetamines on a 100m run would probably minimize time, but other factors will make people consider disallowing such practices.
- Eisenstein 1y ago
  I view Goodhart's law more as a lesson for why we can never achieve a goal by offering specific incentives if we are measuring success by the outcome of the incentives and not by the achievement of the goal. This is of course inevitable if the goal cannot be directly measured but is composed of many constantly moving variables such as education or public health. This doesn't mean we shouldn't bother having such goals, it just means we have to be diligent at pivoting the incentives when it becomes evident that secondary effects are being produced at the expense of the desired effect.
  - godelski 1y ago
    > This is of course inevitable if the goal cannot be directly measured It's worth noting that no goal can be directly measured[0]. I agree with you, this doesn't mean we shouldn't bother with goals. They are fantastic tools. But they are guides. The better aligned our proxy measurement is with the intended measurement then the less we have to interpret our results. We have to think less, spending less energy. But even poorly defined goals can be helpful, as they get refined as we progress in them. We've all done this since we were kids and we do this to this day. All long term goals are updated as we progress in them. It's not like we just state a goal and then hop on the railroad to success. It's like writing tests for code. Tests don't prove that your code is bug free (can't write a test for a bug you don't know about: unknown unknown). But tests are still helpful because they help evidence the code is bug free and constrain the domain in which bugs can live. It's also why TDD is naive, because tests aren't proof and you have to continue to think beyond the tests. [0] https://news.ycombinator.com/item?id=45555551 https://news.ycombinator.com/item?id=45555551

bjornsing 1y ago

Yeah I think this is a general principle. Just look at the quality of US presidents over time, or generations of top physicists. I guess it’s just a numbers game: the number of genuinely interested people is relatively constant while the number of gamers grows with the compensation and perceived status of the activity. So when compensation and perceived status skyrockets the ratio between those numbers changes drastically.

godelski 1y ago
I think the number of generally interested people goes up. Maybe the percent stays the same? But honestly, I think we kill passion for a lot of people. To be cliche, how many people lose the curiosity of a child? I think the cliche exists for a reason. It seems the capacity is in all of us and even once existed.
- bjornsing 1y ago
  To some extent I think that’s just human nature, or even animal nature. The optimal explore / exploit tradeoff changes as we age. When we’re children it’s beneficial to explore. As adults it’s often more beneficial to exploit. But you need cultural and organizational safeguards that protect those of us who are more childish and explorative from those that are more cynical and exploitative. Otherwise pursuits of truth aren’t very fruitful.

crystal_revenge 1y ago

I've spent most of my career working, chatting and hanging out with what might be best described as "passionate weirdos" in various quantitative areas of research. I say "weirdos" because they're people driven by an obsession with a topic, but don't always fit the mold by having the ideal combination of background, credentials and personality to land them on a big tech company research team. The other day I was spending some time with a researcher from Deep Mind and I was surprised to find that while they were sharp and curious to an extent, nearly every ounce of energy they expended on research was strategic. They didn't write about research they were fascinated by, they wrote and researched on topics they strategically felt had the highest probability getting into a major conference in a short period of time to earn them a promotion. While I was a bit disappointed, I certainly didn't judge them because they are just playing the game. This person probably earns more than many rooms of smart, passionate people I've been in, and that money isn't for smarts alone; it's for appealing to the interests of people with the money. You can see this very clearly by comparing the work being done in the LLM space to that being done in the Image/Video diffusion model space. There's much more money in LLMs right now, and the field is flooded with papers on any random topic. If you dive in, most of them are not reproducible or make very questionable conclusions based on the data they present, but that's not of very much concern so long as the paper can be added to a CV. In the stable diffusion world it's mostly people driven by personal interest (usually very non-commericial personal interests) and you see tons of innovation in that field but almost no papers. In fact, if you really want to understand a lot of the most novel work coming out of the image generation world you often need to dig into PRs made by an anonymous users with anime themed profile pic. The bummer of course is that there are very hard limits on what any researcher can do with a home GPU training setup. It does lead to creative solutions to problems, but I can't help but wonder what the world would look like if more of these people had even a fraction of the resources available exclusively to people playing the game.

smokel 1y ago
> I certainly didn't judge them because they are just playing the game. Please do judge them for being parasitical. They might seem successful by certain measures, like the amount of money they make, but I for one simply dislike it when people only think about themselves. As a society, we should be more cautious about narcissism and similar behaviors. Also, in the long run, this kind of behaviour makes them an annoying person at parties.

xvector 1y ago

I have seen absolutely incredible, best in the world type engineers, much smarter than myself, get fired from my FAANG because of the performance games. I persist because I'm fantastic at politics while being good enough to do my job. Feels weird man.

t_serpico 1y ago

But there is no way to know who is truly the 'best'. The people who position and market themselves to be viewed as the best are the only ones who even have a chance to be viewed as such. So if you're a great researcher but don't project yourself that way, no one will ever know you're a great researcher (except for the other great researchers who aren't really invested in communicating how great you are). The system seems to incentivize people to not only optimize for their output but also their image. This isn't a bad thing per se, but is sort of antithetical to the whole shoulder of giants ethos of science.

kcexn 1y ago
The problem is that the best research is not a competitive process but a collaborative one. Positioning research output as a race or a competition is already problematic.
- bwfan123 1y ago
  right. Also, the idea that there is a "best" researcher is already problematic. You could have 10 great people in a team, and it would be hard to rank them. Rating people in order of performance in a team is contradictory to the idea of building a great team. ie, you could have 10 people all rated 10 which is really the goal when building a team.

rightbyte 1y ago

This is an interesting theory. I think there is something to it. It is really hard to do good in a competitive environment. Very constrained.

meindnoch 1y ago

Goodhart's law

RataNova 1y ago

Anytime a system gets hyper-competitive and the stakes are high, it starts selecting for people who are good at playing the system rather than just excelling at the underlying skill

bwfan123 1y ago

I would categorize people into 2 broad extremes. 1) those that care two hoots about what others or the system expects of them and in that sense are authentic and 2) those that only care about what others or the system expects of them, and in that sense are not authentic. There is a spectrum in there.

godelski 1y ago
If you haven't heard this already, you might be interested in Pournelle's Iron Law of Bureaucracy https://jerrypournelle.com/reports/jerryp/iron.html https://jerrypournelle.com/reports/jerryp/iron.html

b00ty4breakfast 1y ago

that's what happens at the top of most competitive domains. Just take a look at pro sports; guys are looking for millimeters to shave off and they turn to "playing the game" rather than merely improving athletic performance. Watching a football game (either kind) and a not-small portion of the action is guys trying to draw penalties or exploit the rules to get an edge.

nathan_compton 1y ago

It is pretty simple - if the rewards are great enough and the objective difficult enough, at some point it becomes more efficient to kneecap your competitors rather than to try to outrun them. I genuinely thing science would be better served if scientist got paid modest salaries to pursue their own research interests and all results became public domain. So many Universities now fancy themselves startup factories, and startups are great for some things, no doubt, but I don't think pure research is always served by this strategy.

godelski 1y ago
> if scientist got paid modest salaries to pursue their own research interests and all results became public domain I would make that deal in a heartbeat[0,1]. We made a mistake by making academia a business. The point was that certain research creates the foundation for others to stand on, but it is difficult to profit off those innovations and by making those innovations public then the society at large will profit by several orders of magnitude more than you would have if you could have. Newton and Leibniz didn't become billionaires by inventing calculus, yet we wouldn't have the trillion dollar businesses and half the technology we have today if they hadn't. You could say the same about Tim Burner Lee's innovation. The idea that we have to justify our research and sell it as profitable is insane. It is as if being unaware of the past itself. Yeah, there's lots of failures in research, it's hard to push the bounds of human knowledge (surprise?). But there are hundreds, if not millions, of examples where that innovation results in so much value that the entire global revenue is not enough. Because the entire global revenue stands on this very foundation. I'm not saying scientists need to be billionaires, but it's fucking ridiculous that we have to fight so hard to justify buying a fucking laptop. It is beyond absurd. [0] https://news.ycombinator.com/item?id=45422828 https://news.ycombinator.com/item?id=45422828 [1] https://news.ycombinator.com/item?id=43959309 https://news.ycombinator.com/item?id=43959309