4 ms·

> Garman is also not keen on another idea about AI – measuring its value by what percentage of code it contributes at an organization. You really want to belie

> Garman is also not keen on another idea about AI – measuring its value by what percentage of code it contributes at an organization. You really want to believe, maybe even need to believe, that anyone who comes up with this idea in their head has never written a single line of code in their life. It is on its face absurd. And yet I don't doubt for a second that Garman et al. have to fend off legions of hacks who froth at the mouth over this kind of thing.

exasperaited 1y ago
All that said, I'm very keen on companies telling me how much of their codebase was written by AI. I just won't use that information in quite the excitable, optimistic way they offer it.
- NoMoreNicksLeft 1y ago
  I want to have the model re-write patent applications, and if any portion of your patent filing was replicated by it your patent is denied as obvious and derivative.
- jihadjihad 1y ago
  "...just raised a $20M Series B and are looking to expand the team and products offered. We are fully bought-in to generative AI — over 40% of our codebase is built and maintained by AI, and we expect this number to continue to grow as the tech evolves and the space matures." "What does your availability over the next couple of weeks look like to chat about this opportunity?"
  - jraph 1y ago
    "Yeah, quite busy over the next couple of weeks actually… the next couple of decades, really - awful how quickly time fills by itself these days, right? I'd have contributed towards lowering that 40% number which seems contrary to your goals anyway. But here's my card, should you need help with debugging something tricky some time in the near future and nobody manages to figure it out internally. I may be able to make room for you if you can afford it. I might be VERY busy though."
rhplus 1y ago
Given the way that a lot of AI coding actually works, it’s like asking what percent of code was written by hitting tab to autocomplete (intellisense) or what percent of a document benefited from spellcheck.
- SV_BubbleTime 1y ago
  While most of us know the next word guessing is how it works in reality… That sentiment ignores the magic of how well this works. There are mind blowing moments using AI coding, to pretend that it’s “just auto correct and tab complete” is just as deceiving as “you can vibe code complete programs”.
  - rhplus 1y ago
    That’s not my point. I’m saying that half my code could be AI assisted, but counting which LOC were AI and which were mine would be near impossible.
Buttons840 1y ago
Time to apply the best analogy I've ever heard. > "Measuring software productivity by lines of code is like measuring progress on an airplane by how much it weighs." -- Bill Gates Do we reward the employee who has added the most weight? Do we celebrate when the AI has added a lot of weight? At first, it seems like, no, we shouldn't, but actually, it depends. If a person or AI is adding a lot of weight, but it is really important weight, like the engines or the main structure of the plane, then yeah, even though it adds a lot of weight, it's still doing genuinely impressive work. A heavy airplane is more impressive than a light weight one (usually).
- subhro 1y ago
  I just can’t resist myself when airplanes come up in discussion. I completely understand your analogy and you are right. However just to nitpick, it is actually super important to have a weight on the airplane at the right place. You have to make sure that your aeroplane does not become tail heavy or it is not recoverable from a stall. Also a heavier aeroplane, within its gross weight, is actually safer as the safe manoeuverable speed increases with weight.
  - RugnirViking 1y ago
    right, thats why its a great analogy - because you also need to have at least some code in a successful piece of software. But simply measuring by the amount of code leads to weird and perverse incentives - code added without thought is not good, and too much code can itself be a problem. Of course, the literal balancing aspect isn't as important.
  - fny 1y ago
    I think this makes the analogy even more apt. If someone adds more code to the wrong places for the sake of adding more code, the software may not be recoverable for future changes or from bugs. You also often need to add code in the right places for robustness.
  - dahart 1y ago
    > a heavier aeroplane … is actually safer Just to nitpick your nitpick, that’s only true up to a point, and the range of safe weights isn’t all that big really - max payload on most planes is a fraction of the empty weight. And planes can be overweight, reducing weight is a good thing and perhaps needed far more often than adding weight is needed. The point of the analogy was that over a certain weight, the plane doesn’t fly at all. If progress on a plane is safety, stability, or speed, we can measure those things directly. If weight distribution is important to those, that’s great we can measure weight and distribution in service of stability, but weight isn’t the primary thing we use. Like with airplane weight, you absolutely need some code to get something done, and sometimes more is better. But is more better as a rule? Absolutely not.
techpineapple 1y ago
Something I wonder about the percent of code - I remember like 5-10 years ago there was a series of articles about Google generating a lot of their code programmatically, I wonder if they just adapted their code gen to AI. I bet Google has a lot of tools to say convert a library from one language to another or generate a library based on an API spec. The 30% of code these LLMs are supposedly writing is probably in this camp, not net novel new features.
- kreyenborgi 1y ago
  Is that why gmail loads so slowly these days
DanielHB 1y ago
It is like measuring company output based on stuff done through codegen...
ks2048 1y ago
When I see these stats, I think of all the ways "percentage of code" could be defined. I ask an AI 4 times to write a method for me. After it keeps failing, I just write it myself. AI wrote 80% of the code!
shit_game 1y ago
This reminds me of a piece on folklore.org by Andy Hertzfeld[0], regarding Bill Atkinson. A "KPI" was introduced at Apple in which engineers were required to report how many lines of code they had written over the week. Bill (allegedly) claimed "-2000" (a completely, astonishingly negative report), and supposedly the managers reconsidered the validity of the "KPI" and stopped using it. I don't know how true this is in fact, but I do know how true this is in my work - you cannot apply some arbitrary "make the number bigger" goal to everything and expect it to improve anything. It feels a bit weird seeing "write more lines of code" becoming a key metric again. It never worked, and is damn-near provably never going to work. The value of source code is not in any way tied to its quantity, but value still proves hard to quantify, 40 years later. 0. https://www.folklore.org/Negative_2000_Lines_Of_Code.html https://www.folklore.org/Negative_2000_Lines_Of_Code.html
- wat10000 1y ago
  Goodhart's law: when a measure becomes a target, it ceases to be a good measure.
ozgrakkurt 1y ago
It is a really attractive idea for lazy people who don’t want to learn things

exasperaited 1y ago

All that said, I'm very keen on companies telling me how much of their codebase was written by AI. I just won't use that information in quite the excitable, optimistic way they offer it.

NoMoreNicksLeft 1y ago
I want to have the model re-write patent applications, and if any portion of your patent filing was replicated by it your patent is denied as obvious and derivative.
jihadjihad 1y ago
"...just raised a $20M Series B and are looking to expand the team and products offered. We are fully bought-in to generative AI — over 40% of our codebase is built and maintained by AI, and we expect this number to continue to grow as the tech evolves and the space matures." "What does your availability over the next couple of weeks look like to chat about this opportunity?"
- jraph 1y ago
  "Yeah, quite busy over the next couple of weeks actually… the next couple of decades, really - awful how quickly time fills by itself these days, right? I'd have contributed towards lowering that 40% number which seems contrary to your goals anyway. But here's my card, should you need help with debugging something tricky some time in the near future and nobody manages to figure it out internally. I may be able to make room for you if you can afford it. I might be VERY busy though."

rhplus 1y ago

Given the way that a lot of AI coding actually works, it’s like asking what percent of code was written by hitting tab to autocomplete (intellisense) or what percent of a document benefited from spellcheck.

SV_BubbleTime 1y ago
While most of us know the next word guessing is how it works in reality… That sentiment ignores the magic of how well this works. There are mind blowing moments using AI coding, to pretend that it’s “just auto correct and tab complete” is just as deceiving as “you can vibe code complete programs”.
- rhplus 1y ago
  That’s not my point. I’m saying that half my code could be AI assisted, but counting which LOC were AI and which were mine would be near impossible.

Buttons840 1y ago

Time to apply the best analogy I've ever heard. > "Measuring software productivity by lines of code is like measuring progress on an airplane by how much it weighs." -- Bill Gates Do we reward the employee who has added the most weight? Do we celebrate when the AI has added a lot of weight? At first, it seems like, no, we shouldn't, but actually, it depends. If a person or AI is adding a lot of weight, but it is really important weight, like the engines or the main structure of the plane, then yeah, even though it adds a lot of weight, it's still doing genuinely impressive work. A heavy airplane is more impressive than a light weight one (usually).

subhro 1y ago
I just can’t resist myself when airplanes come up in discussion. I completely understand your analogy and you are right. However just to nitpick, it is actually super important to have a weight on the airplane at the right place. You have to make sure that your aeroplane does not become tail heavy or it is not recoverable from a stall. Also a heavier aeroplane, within its gross weight, is actually safer as the safe manoeuverable speed increases with weight.
- RugnirViking 1y ago
  right, thats why its a great analogy - because you also need to have at least some code in a successful piece of software. But simply measuring by the amount of code leads to weird and perverse incentives - code added without thought is not good, and too much code can itself be a problem. Of course, the literal balancing aspect isn't as important.
- fny 1y ago
  I think this makes the analogy even more apt. If someone adds more code to the wrong places for the sake of adding more code, the software may not be recoverable for future changes or from bugs. You also often need to add code in the right places for robustness.
- dahart 1y ago
  > a heavier aeroplane … is actually safer Just to nitpick your nitpick, that’s only true up to a point, and the range of safe weights isn’t all that big really - max payload on most planes is a fraction of the empty weight. And planes can be overweight, reducing weight is a good thing and perhaps needed far more often than adding weight is needed. The point of the analogy was that over a certain weight, the plane doesn’t fly at all. If progress on a plane is safety, stability, or speed, we can measure those things directly. If weight distribution is important to those, that’s great we can measure weight and distribution in service of stability, but weight isn’t the primary thing we use. Like with airplane weight, you absolutely need some code to get something done, and sometimes more is better. But is more better as a rule? Absolutely not.

techpineapple 1y ago

Something I wonder about the percent of code - I remember like 5-10 years ago there was a series of articles about Google generating a lot of their code programmatically, I wonder if they just adapted their code gen to AI. I bet Google has a lot of tools to say convert a library from one language to another or generate a library based on an API spec. The 30% of code these LLMs are supposedly writing is probably in this camp, not net novel new features.

kreyenborgi 1y ago
Is that why gmail loads so slowly these days

DanielHB 1y ago

It is like measuring company output based on stuff done through codegen...

ks2048 1y ago

When I see these stats, I think of all the ways "percentage of code" could be defined. I ask an AI 4 times to write a method for me. After it keeps failing, I just write it myself. AI wrote 80% of the code!

shit_game 1y ago

This reminds me of a piece on folklore.org by Andy Hertzfeld[0], regarding Bill Atkinson. A "KPI" was introduced at Apple in which engineers were required to report how many lines of code they had written over the week. Bill (allegedly) claimed "-2000" (a completely, astonishingly negative report), and supposedly the managers reconsidered the validity of the "KPI" and stopped using it. I don't know how true this is in fact, but I do know how true this is in my work - you cannot apply some arbitrary "make the number bigger" goal to everything and expect it to improve anything. It feels a bit weird seeing "write more lines of code" becoming a key metric again. It never worked, and is damn-near provably never going to work. The value of source code is not in any way tied to its quantity, but value still proves hard to quantify, 40 years later. 0. https://www.folklore.org/Negative_2000_Lines_Of_Code.html https://www.folklore.org/Negative_2000_Lines_Of_Code.html

wat10000 1y ago
Goodhart's law: when a measure becomes a target, it ceases to be a good measure.

ozgrakkurt 1y ago

It is a really attractive idea for lazy people who don’t want to learn things