3 ms·

It's actually incredible the extent to which non devs imposing KPIs on devs underestimate how badly this will get gamed, whether it's AIs, PR/line counting or w

It's actually incredible the extent to which non devs imposing KPIs on devs underestimate how badly this will get gamed, whether it's AIs, PR/line counting or whatever.

celdon25 6mo ago
Easily fixable with another KPI to measure the gaming itself :P
ambicapter 6mo ago
Someone at my job uses AI tools to reformat his code...
- i_love_retros 6mo ago
  My coworker said he does that too. Also have coworkers using AI to run git commands. Nothing fancy either- just pull, push, merge etc
  - fragmede 6mo ago
    When Claude says "Shall I push it", it's way easier to just respond "yes" than it is to open a new terminal and run git push, and if you're being graded on how much AI tokens you use, saying yes looks even better for your metrics!
  - phainopepla2 6mo ago
    Do you mean they tell an LLM "push this code", or that the LLM runs the commands as part of a larger agentic workflow. The former is crazy but the latter makes sense.
    - i_love_retros 6mo ago
      Former. Pull main and merge into my branch, resolve conflicts, etc (Yes I forgot, they have the agent resolve git conflicts!)
- SatvikBeri 6mo ago
  I actually do this, but that's mostly because our team reviewed all the existing autoformatters for the relatively obscure language we use, and either really hated the formatting or found that they actually introduced errors!
darth_avocado 6mo ago
Gaming is one thing, fundamentally not understanding how engineering works will lead to shittier outcomes and cost the company in ways the management will never understand. Management in the age of AI is falling for the doorman fallacy wrt engineering. If lines of code were the most valuable aspect of software engineering, my front end JavaScript intern would’ve been the most valuable person in the company. https://www.jaakkoj.com/concepts/doorman-fallacy https://www.jaakkoj.com/concepts/doorman-fallacy
- hnthrow0287345 6mo ago
  >Gaming is one thing, fundamentally not understanding how engineering works will lead to shittier outcomes and cost the company in ways the management will never understand. That means nothing to them: they jump ship and find another job just like devs do. The whole industry has been musical chairs for a while.
jimbokun 6mo ago
I think PRs is pretty good, IF 1. you sample a few to see that they are actually meaningful, 2. they go to prod and are validated without having to roll back. Still needs to be managed. But it should be much easier for a manager to catch an engineer gaming PRs than something like AI use or lines of code.
- dieortin 6mo ago
  It’s very easy to split changes in more PRs than needed to boost the number.
joshuastuden 6mo ago
Exactly. At Cerebras I know of several people who burn tokens on completely USELESS tasks (randomly changing pixels in an image) just to keep them high up on the token leaderboard. I suspect the other tokenboard leaders are doing the same. They made the metric "token usage" (which is just a proxy for LOC) so that's what they're gonna get.

It's actually incredible the extent to which non devs imposing KPIs on devs underestimate how badly this will get gamed, whether it's AIs, PR/line counting or whatever.

celdon25 6mo ago

Easily fixable with another KPI to measure the gaming itself :P

ambicapter 6mo ago

Someone at my job uses AI tools to reformat his code...

i_love_retros 6mo ago
My coworker said he does that too. Also have coworkers using AI to run git commands. Nothing fancy either- just pull, push, merge etc
- fragmede 6mo ago
  When Claude says "Shall I push it", it's way easier to just respond "yes" than it is to open a new terminal and run git push, and if you're being graded on how much AI tokens you use, saying yes looks even better for your metrics!
- phainopepla2 6mo ago
  Do you mean they tell an LLM "push this code", or that the LLM runs the commands as part of a larger agentic workflow. The former is crazy but the latter makes sense.
  - i_love_retros 6mo ago
    Former. Pull main and merge into my branch, resolve conflicts, etc (Yes I forgot, they have the agent resolve git conflicts!)
SatvikBeri 6mo ago
I actually do this, but that's mostly because our team reviewed all the existing autoformatters for the relatively obscure language we use, and either really hated the formatting or found that they actually introduced errors!

darth_avocado 6mo ago

Gaming is one thing, fundamentally not understanding how engineering works will lead to shittier outcomes and cost the company in ways the management will never understand. Management in the age of AI is falling for the doorman fallacy wrt engineering. If lines of code were the most valuable aspect of software engineering, my front end JavaScript intern would’ve been the most valuable person in the company. https://www.jaakkoj.com/concepts/doorman-fallacy https://www.jaakkoj.com/concepts/doorman-fallacy

hnthrow0287345 6mo ago
>Gaming is one thing, fundamentally not understanding how engineering works will lead to shittier outcomes and cost the company in ways the management will never understand. That means nothing to them: they jump ship and find another job just like devs do. The whole industry has been musical chairs for a while.

jimbokun 6mo ago

I think PRs is pretty good, IF 1. you sample a few to see that they are actually meaningful, 2. they go to prod and are validated without having to roll back. Still needs to be managed. But it should be much easier for a manager to catch an engineer gaming PRs than something like AI use or lines of code.

dieortin 6mo ago
It’s very easy to split changes in more PRs than needed to boost the number.

joshuastuden 6mo ago

Exactly. At Cerebras I know of several people who burn tokens on completely USELESS tasks (randomly changing pixels in an image) just to keep them high up on the token leaderboard. I suspect the other tokenboard leaders are doing the same. They made the metric "token usage" (which is just a proxy for LOC) so that's what they're gonna get.