4 ms·

"The leaderboard, which ranked employees and teams by token consumption, inadvertently incentivized usage volume over productive output." Who could possibly ha

by simonw 4mo ago

"The leaderboard, which ranked employees and teams by token consumption, inadvertently incentivized usage volume over productive output." Who could possibly have predicted that happening?

qwertytyyuu 4mo ago
I know right? What did the leadership think would happen when they give some of the worlds greatest software engineers (supportably), a easily quantifiable metric to target?
- VygmraMGVl 4mo ago
  The leaderboard wasn't leadership generated, it was engineer generated from internally available data. The leadership target is "impact" from ai tools.
  - Avicebron 4mo ago
    Budget impact is technically impact.
  - gtowey 4mo ago
    What a wonderful scapegoat! Technically it's all "engineer created" because the managers generally don't do technical work. I bet many managers pushed their reports to increase their usage during their 1:1 meetings based on data from the leaderboard. If management had any sense that it was a bad metric, they had ample time to get ahead of it and take it down and provide appropriate guidance. Instead, predictably, they waited until it was a full on disaster and a crisis before acting.
    - VygmraMGVl 4mo ago
      At one point there were over 70 different token maxing dashboards as the management had a game of whack a mole trying to remove them. There definitely was encouragement from management about a year ago to increase ai usage, but once Claude code was allowed, they didn't really need to encourage anyone any more.
jghn 4mo ago
> Who could possibly have predicted that happening? Charles Goodhart :-)
- goldenarm 4mo ago
  https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
sharts 4mo ago
How dare you question the most effective allocators of capital.
0cf8612b2e1e 4mo ago
Now come on, there was a recent post where the author argued that infallible management knew this would happen, but was part of the double-secret-probation strategy to get the cogs to finally start using AI.
- SpicyLemonZest 4mo ago
  I still think this is true and it’s not obvious to me from the source article that Meta believes otherwise. I couldn’t find the full memo, do they claim the leaderboard or “tokenmaxxing” era was a mistake?
  - TheOtherHobbes 4mo ago
    Would they admit it if it was? Or would they try to find a plausible rationalisation for wasting billions without any return?
Aurornis 4mo ago
A past employer thought it was a good idea to put up a leaderboard of who sent the most Slack messages. They celebrated the people at the top for being so active. Predictably, everyone started talking in Slack like their jobs depended on it. Everyone was responding to everything. Instead of writing out a complete message and pressing enter, they'd send each fragment of the sentence as a new line. The Slack leaderboard was never shown again. Unfortunately the habit remained because people were afraid they were going to be secretly judged by how much Slack activity they generated. I expect the same thing is going to happen at companies who had token leaderboards. Once you've instilled that fear in people, they internalize the expectation.
- Loughla 4mo ago
  You have to realize that if you set a measure, you're actually setting a goal for your employees. There is no such thing as a meaningless metric; why else would you measure it? No amount of "this isn't used for anything" will change that. It's inherent in human nature in the 21st century to believe any and all metrics will be used against them, and therefore must be gamed. It's why you also have to set UNBELIEVABLY clear goals and have incentives tied to those goals. Incentives meaning money. If you want to measure things, measure them. But have clear, consistent, and meaningful goals tied to bonuses or something if you want a thing done correctly.
  - 4yfr 4mo ago
    Kinda. The answer is simpler on the surface: focus. Generally the problem is the larger the firm’s operations, the harder it is to focus. Apple is the only firm that has done well on this consistently and doesn’t have a huge grave yard of failures to show for it.
    - estearum 4mo ago
      Are you saying what people are hoping to achieve with stupid goals? Because yeah, obviously. But the point is that they're stupid, so they don't achieve that, and that failure is 100% knowable in most scenarios.
dzonga 4mo ago
unfortunately at big tech, this shit will keep happening. people who make it to managers tend to have bozo tendencies & are yes men. before it was lines of code, Jira tickets closed. Now it's tokens spent.
darth_avocado 4mo ago
> Who could possibly have predicted that happening? Everyone except the executives who get paid millions to predict exactly that.
- Avicebron 4mo ago
  Not a problem. There are thousands of employees standing by, willing to sacrifice their jobs for their vision. It's a hard job, someone has to not pay consequences for bad decisions.
- red-iron-pine 4mo ago
  presumably they covered this in the boilerplate MBA courses they took, non?
zulux 4mo ago
[flagged]
deleted 4mo ago
[deleted]
ryanschaefer 4mo ago
It’s funny how many times the same thing happens at each large company. I think people’s thought process is this: > Oh wow! If I paid for this myself I would have spent a lot of money! Are other people spending as much as me? I’m going to create a leaderboard! > Oh no, my misinformed manager is using the leaderboard as a slight of hand for work. I need to game this now. Then the leaderboard is banned… I can’t see how this ever really goes up the chain beyond director.
- what 4mo ago
  There is zero chance that this is how the leaderboards came to be.
  - ryanschaefer 4mo ago
    What makes you assert that?
skizm 4mo ago
It wasn't leadership doing this though. Any meta IC can generate internal apps and dashboards. This was unofficial and unsupported. Some random IC just made it for fun. Management is usually pretty lax with stuff like this (plenty of games and joke internal apps) so they left it up until it became a problem.
John23832 4mo ago
Given that Meta has run 5ish layoffs at this point, and everyone is in survival mode, what did they expect? Everyone wants to juice whatever numbers possible to keep their jobs.
giancarlostoro 4mo ago
I still don't understand how Mark Zuckerberg has any serious investors, he went on this AI tangent and has absolutely nothing to show for it, despite FB / Meta having built some key tech in the space. He needs to stop trying to do something "different" and literally try and build a serious coding agent he can sell, he could have probably had something worthwhile in that space by now. He started being drastically more serious into AI in 2022, and 2023 and he has nothing to show for it. Heck, he could have rented GPUs the way Elon did at this point and either mended the bleeding or stopped it, not sure how many he has, but it beats losing this badly. If he doesn't wake up and learn how to business, I suspect he will lose his empire he's built up for himself.
- MangoCoffee 4mo ago
  >he could have rented GPUs the way Elon did at this point "Meta building cloud business to sell excess AI capacity, Bloomberg News reports Meta building cloud business to sell excess AI capacity, Bloomberg News reports" https://www.reuters.com/business/meta-sell-excess-ai-computing-capacity-via-cloud-business-bloomberg-news-reports-2026-07-01/ https://www.reuters.com/business/meta-sell-excess-ai-computi...
  - giancarlostoro 4mo ago
    That's fair, I guess still having that sort of leaderboard is a foolish way to burn a billion dollars.
- cadamsdotcom 4mo ago
  Haven’t got numbers so I might be wrong, but I suspect it is dwarfed by the present size and future potential of Meta’s ads business.
thewhitetulip 4mo ago
Even in firms that don't hold this board, giving unlimited AI to people means those who don't bother to learn technology can now deliver 100x their capacity, in very poor quality code
nsonha 4mo ago
My company has an AI leaderboard and ONE ranking to be this, AND OTHER rankings like efficiency (loc merged per token). No one is so stupid that they think any single one of this is to be optimized (gamed) for. Tech journalists have low opinion of people with actual skills who actually contribute to society, and when their opinions get posted here, it's often selectively echoed by people looking for a reason to feel smarter than the industry.
- judahmeek 4mo ago
  Loc merged per token seems like an even worse metric because it punishes tokens used for refactoring, brainstorming, & code review
  - deleted 4mo ago
    [deleted]

by simonw 4mo ago

"The leaderboard, which ranked employees and teams by token consumption, inadvertently incentivized usage volume over productive output." Who could possibly have predicted that happening?

qwertytyyuu 4mo ago

I know right? What did the leadership think would happen when they give some of the worlds greatest software engineers (supportably), a easily quantifiable metric to target?

VygmraMGVl 4mo ago
The leaderboard wasn't leadership generated, it was engineer generated from internally available data. The leadership target is "impact" from ai tools.
- Avicebron 4mo ago
  Budget impact is technically impact.
- gtowey 4mo ago
  What a wonderful scapegoat! Technically it's all "engineer created" because the managers generally don't do technical work. I bet many managers pushed their reports to increase their usage during their 1:1 meetings based on data from the leaderboard. If management had any sense that it was a bad metric, they had ample time to get ahead of it and take it down and provide appropriate guidance. Instead, predictably, they waited until it was a full on disaster and a crisis before acting.
  - VygmraMGVl 4mo ago
    At one point there were over 70 different token maxing dashboards as the management had a game of whack a mole trying to remove them. There definitely was encouragement from management about a year ago to increase ai usage, but once Claude code was allowed, they didn't really need to encourage anyone any more.

jghn 4mo ago

> Who could possibly have predicted that happening? Charles Goodhart :-)

goldenarm 4mo ago
https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law

sharts 4mo ago

How dare you question the most effective allocators of capital.

0cf8612b2e1e 4mo ago

Now come on, there was a recent post where the author argued that infallible management knew this would happen, but was part of the double-secret-probation strategy to get the cogs to finally start using AI.

SpicyLemonZest 4mo ago
I still think this is true and it’s not obvious to me from the source article that Meta believes otherwise. I couldn’t find the full memo, do they claim the leaderboard or “tokenmaxxing” era was a mistake?
- TheOtherHobbes 4mo ago
  Would they admit it if it was? Or would they try to find a plausible rationalisation for wasting billions without any return?

Aurornis 4mo ago

A past employer thought it was a good idea to put up a leaderboard of who sent the most Slack messages. They celebrated the people at the top for being so active. Predictably, everyone started talking in Slack like their jobs depended on it. Everyone was responding to everything. Instead of writing out a complete message and pressing enter, they'd send each fragment of the sentence as a new line. The Slack leaderboard was never shown again. Unfortunately the habit remained because people were afraid they were going to be secretly judged by how much Slack activity they generated. I expect the same thing is going to happen at companies who had token leaderboards. Once you've instilled that fear in people, they internalize the expectation.

Loughla 4mo ago
You have to realize that if you set a measure, you're actually setting a goal for your employees. There is no such thing as a meaningless metric; why else would you measure it? No amount of "this isn't used for anything" will change that. It's inherent in human nature in the 21st century to believe any and all metrics will be used against them, and therefore must be gamed. It's why you also have to set UNBELIEVABLY clear goals and have incentives tied to those goals. Incentives meaning money. If you want to measure things, measure them. But have clear, consistent, and meaningful goals tied to bonuses or something if you want a thing done correctly.
- 4yfr 4mo ago
  Kinda. The answer is simpler on the surface: focus. Generally the problem is the larger the firm’s operations, the harder it is to focus. Apple is the only firm that has done well on this consistently and doesn’t have a huge grave yard of failures to show for it.
  - estearum 4mo ago
    Are you saying what people are hoping to achieve with stupid goals? Because yeah, obviously. But the point is that they're stupid, so they don't achieve that, and that failure is 100% knowable in most scenarios.

dzonga 4mo ago

unfortunately at big tech, this shit will keep happening. people who make it to managers tend to have bozo tendencies & are yes men. before it was lines of code, Jira tickets closed. Now it's tokens spent.

darth_avocado 4mo ago

> Who could possibly have predicted that happening? Everyone except the executives who get paid millions to predict exactly that.

Avicebron 4mo ago
Not a problem. There are thousands of employees standing by, willing to sacrifice their jobs for their vision. It's a hard job, someone has to not pay consequences for bad decisions.
red-iron-pine 4mo ago
presumably they covered this in the boilerplate MBA courses they took, non?

zulux 4mo ago

[flagged]

deleted 4mo ago

[deleted]

ryanschaefer 4mo ago

It’s funny how many times the same thing happens at each large company. I think people’s thought process is this: > Oh wow! If I paid for this myself I would have spent a lot of money! Are other people spending as much as me? I’m going to create a leaderboard! > Oh no, my misinformed manager is using the leaderboard as a slight of hand for work. I need to game this now. Then the leaderboard is banned… I can’t see how this ever really goes up the chain beyond director.

what 4mo ago
There is zero chance that this is how the leaderboards came to be.
- ryanschaefer 4mo ago
  What makes you assert that?

skizm 4mo ago

It wasn't leadership doing this though. Any meta IC can generate internal apps and dashboards. This was unofficial and unsupported. Some random IC just made it for fun. Management is usually pretty lax with stuff like this (plenty of games and joke internal apps) so they left it up until it became a problem.

John23832 4mo ago

Given that Meta has run 5ish layoffs at this point, and everyone is in survival mode, what did they expect? Everyone wants to juice whatever numbers possible to keep their jobs.

giancarlostoro 4mo ago

I still don't understand how Mark Zuckerberg has any serious investors, he went on this AI tangent and has absolutely nothing to show for it, despite FB / Meta having built some key tech in the space. He needs to stop trying to do something "different" and literally try and build a serious coding agent he can sell, he could have probably had something worthwhile in that space by now. He started being drastically more serious into AI in 2022, and 2023 and he has nothing to show for it. Heck, he could have rented GPUs the way Elon did at this point and either mended the bleeding or stopped it, not sure how many he has, but it beats losing this badly. If he doesn't wake up and learn how to business, I suspect he will lose his empire he's built up for himself.

MangoCoffee 4mo ago
>he could have rented GPUs the way Elon did at this point "Meta building cloud business to sell excess AI capacity, Bloomberg News reports Meta building cloud business to sell excess AI capacity, Bloomberg News reports" https://www.reuters.com/business/meta-sell-excess-ai-computing-capacity-via-cloud-business-bloomberg-news-reports-2026-07-01/ https://www.reuters.com/business/meta-sell-excess-ai-computi...
- giancarlostoro 4mo ago
  That's fair, I guess still having that sort of leaderboard is a foolish way to burn a billion dollars.
cadamsdotcom 4mo ago
Haven’t got numbers so I might be wrong, but I suspect it is dwarfed by the present size and future potential of Meta’s ads business.

thewhitetulip 4mo ago

Even in firms that don't hold this board, giving unlimited AI to people means those who don't bother to learn technology can now deliver 100x their capacity, in very poor quality code

nsonha 4mo ago

My company has an AI leaderboard and ONE ranking to be this, AND OTHER rankings like efficiency (loc merged per token). No one is so stupid that they think any single one of this is to be optimized (gamed) for. Tech journalists have low opinion of people with actual skills who actually contribute to society, and when their opinions get posted here, it's often selectively echoed by people looking for a reason to feel smarter than the industry.

judahmeek 4mo ago
Loc merged per token seems like an even worse metric because it punishes tokens used for refactoring, brainstorming, & code review
- deleted 4mo ago
  [deleted]