5 ms·

I believe that quote in Thomas’ blog can be attributed to me. I’ve at least said something near enough to him that I don’t mind claiming it. I _never_ made the

I believe that quote in Thomas’ blog can be attributed to me. I’ve at least said something near enough to him that I don’t mind claiming it. I _never_ made the claim that you could call that 10x productivity improvement. I’m hesitant to categorize productivity in software in numeric terms as it’s such a nuanced concept. But I’ll stand by my impression that a developer using ai tools will generate code at a perceptibly faster pace than one who isn’t. I mentioned in another comment the major flaw in your productivity calculation, is that you aren’t accounting for the work that wouldn’t have gotten done otherwise. That’s where my improvements are almost universally coming from. I can improve the codebase in ways that weren’t justifiable before in places that do not suffer from the coordination costs you rightly point out. I no longer feel like my peers are standing still, because they’ve nearly uniformly adopted ai tools. And again, you rightly point out, there isn’t much of a learning curve. If you could develop before them you can figure out how to improve with them. I found it easier than learning vim. As for hallucinations I don’t experience them effectively _ever_. And I do let agents mess with terraform code (in code bases where I can prevent state manipulation or infrastructure changes outside of the agents control). I don’t have any hints on how. I’m using a pretty vanilla Claude code setup. But im not sure how an agent that can write and run compile/test loops could hallucinate.

claytongulick 1y ago
> But I’ll stand by my impression that a developer using ai tools will generate code at a perceptibly faster pace than one who isn’t. And this is the problem. Masterful developers are the ones you pay to reduce lines of code, not create them.
- kasey_junk 1y ago
  Every. Single. Time. You say you get productivity gains from ai tools on the internet someone will tell you that you weren’t good at your job before the ai tooling. Perhaps, start from the assumption that I have in fact spent a fair bit of time doing this job at a high level. Where does that mental exercise take you with regard to your own position on ai tools. In fact, you don’t have to assume I’m qualified to speak on the subject. Your retort assumes that _everyone_ who gets improvement is bad at this. Assume any random proponent isn’t.
  - claytongulick 1y ago
    It's a commentary on one of the things I perceive as a flaw with LLMs, not you. One of the most valuable qualities of humans is laziness. We're constantly seeking efficiency gains, because who wants to carry buckets of water, or take laundry down to the river? Skilled developers excel at this. They are "lazy" when they code - they plan for the future, they construct code in a way that will make their life better, and easier. LLMs don't have this motivation. They will gleefully spit out 1000 lines of code when 10 will do. It's a fundamental flaw.
    - kasey_junk 1y ago
      Now, go back and contemplate what my feedback means if I am well versed on Larry Wall-isms.
  - bit1993 1y ago
    I think what GP is saying is that in most cases generating allot of code is not a good thing. Every line of LLM generated code has to be audited because they are prone to hallucinations and auditing someone else's code is much more difficult and time consuming than auditing your own code. Allot of code also requires more maintenance.
    - tptacek 1y ago
      The comment is premised on the idea that Kasey either doesn't know what a "masterful developer" is or needs to be corrected back to it.
coltonv 1y ago
Appreciate the comment! > I mentioned in another comment the major flaw in your productivity calculation, is that you aren’t accounting for the work that wouldn’t have gotten done otherwise. That’s where my improvements are almost universally coming from. I can improve the codebase in ways that weren’t justifiable before in places that do not suffer from the coordination costs you rightly point out. I'm a bit confused by this. There is work that apparently is unlocking big productivity boosts but was somehow not justified before? Are you referring to places like my ESLint rule example, where eliminating the startup costs of learning how to write one allows you to do things you wouldn't have previously bothered with? If so, I feel like I covered this pretty well in the article and we probably largely agree on the value that productivity boost. My point is still stands that that doesn't scale. If this is not what you mean, feel free to correct me. Appreciate your thoughts on hallucinations. My guess is the difference between what we're experiencing is that in your code hallucinations are still happening but getting corrected after tests are run, whereas my agents typically get stuck in these write-and-test loops and can't figure out how to solve the problem, or it "solves" it by deleting the tests or something like that. I've seen videos and viewed open source AI PRs which end up in similar loops as to what I've experienced, so I think what I see is common. Perhaps that's an indication of that we're trying to solve different problems with agents, or using different languages/libraries, and that explains the divergence of experiences. Either way, I still contend that this kind of productivity boost is likely going to be hard to scale and will get tougher to realize as time goes on. If you keep seeing it, I'd really love to hear more about your methods to see what I'm missing. One thing that has been frustrating me is that people rarely share their workflows after makign big claims. This is unlike previous hype cycles where people would share descriptions of exactly what they did ("we rewrote in Rust, here's how we did it", etc.) Feel free to email me at the address in my about page[1] or send me a request on LinkedIn or whatever. I'm being 100% genuine that I'd love to learn from you! [1] https://colton.dev/about/ https://colton.dev/about/
- kasey_junk 1y ago
  > but getting corrected after tests are run, whereas my agents typically get stuck in these write-and-test loops This maybe a definition problem then. I don’t think “the agent did a dumb thing that it can’t reason out of” is a hallucination. To me a hallucination is a pretty specific failure mode, it invents something that doesn’t exist. Models still do that for me but the build test loop sets them aright on that nearly perfectly. So I guess the model is still hallucinating but the agent isn’t so the output is unimpacted. So I don’t care. For the agent is dumb scenario, I aggressively delete and reprompt. This is something I’ve actually gotten much better at with time and experience, both so it doesn’t happen often and I can course correct quickly. I find it works nearly as well for teaching me about the problem domain as my own mistakes do but is much faster to get to. But if I were going to be pithy. Aggressively deleting work output from an agent is part of their value proposition. They don’t get offended and they don’t need explanations why. Of course they don’t learn well either, that’s on you.

claytongulick 1y ago

> But I’ll stand by my impression that a developer using ai tools will generate code at a perceptibly faster pace than one who isn’t. And this is the problem. Masterful developers are the ones you pay to reduce lines of code, not create them.

kasey_junk 1y ago
Every. Single. Time. You say you get productivity gains from ai tools on the internet someone will tell you that you weren’t good at your job before the ai tooling. Perhaps, start from the assumption that I have in fact spent a fair bit of time doing this job at a high level. Where does that mental exercise take you with regard to your own position on ai tools. In fact, you don’t have to assume I’m qualified to speak on the subject. Your retort assumes that _everyone_ who gets improvement is bad at this. Assume any random proponent isn’t.
- claytongulick 1y ago
  It's a commentary on one of the things I perceive as a flaw with LLMs, not you. One of the most valuable qualities of humans is laziness. We're constantly seeking efficiency gains, because who wants to carry buckets of water, or take laundry down to the river? Skilled developers excel at this. They are "lazy" when they code - they plan for the future, they construct code in a way that will make their life better, and easier. LLMs don't have this motivation. They will gleefully spit out 1000 lines of code when 10 will do. It's a fundamental flaw.
  - kasey_junk 1y ago
    Now, go back and contemplate what my feedback means if I am well versed on Larry Wall-isms.
- bit1993 1y ago
  I think what GP is saying is that in most cases generating allot of code is not a good thing. Every line of LLM generated code has to be audited because they are prone to hallucinations and auditing someone else's code is much more difficult and time consuming than auditing your own code. Allot of code also requires more maintenance.
  - tptacek 1y ago
    The comment is premised on the idea that Kasey either doesn't know what a "masterful developer" is or needs to be corrected back to it.

coltonv 1y ago

Appreciate the comment! > I mentioned in another comment the major flaw in your productivity calculation, is that you aren’t accounting for the work that wouldn’t have gotten done otherwise. That’s where my improvements are almost universally coming from. I can improve the codebase in ways that weren’t justifiable before in places that do not suffer from the coordination costs you rightly point out. I'm a bit confused by this. There is work that apparently is unlocking big productivity boosts but was somehow not justified before? Are you referring to places like my ESLint rule example, where eliminating the startup costs of learning how to write one allows you to do things you wouldn't have previously bothered with? If so, I feel like I covered this pretty well in the article and we probably largely agree on the value that productivity boost. My point is still stands that that doesn't scale. If this is not what you mean, feel free to correct me. Appreciate your thoughts on hallucinations. My guess is the difference between what we're experiencing is that in your code hallucinations are still happening but getting corrected after tests are run, whereas my agents typically get stuck in these write-and-test loops and can't figure out how to solve the problem, or it "solves" it by deleting the tests or something like that. I've seen videos and viewed open source AI PRs which end up in similar loops as to what I've experienced, so I think what I see is common. Perhaps that's an indication of that we're trying to solve different problems with agents, or using different languages/libraries, and that explains the divergence of experiences. Either way, I still contend that this kind of productivity boost is likely going to be hard to scale and will get tougher to realize as time goes on. If you keep seeing it, I'd really love to hear more about your methods to see what I'm missing. One thing that has been frustrating me is that people rarely share their workflows after makign big claims. This is unlike previous hype cycles where people would share descriptions of exactly what they did ("we rewrote in Rust, here's how we did it", etc.) Feel free to email me at the address in my about page[1] or send me a request on LinkedIn or whatever. I'm being 100% genuine that I'd love to learn from you! [1] https://colton.dev/about/ https://colton.dev/about/

kasey_junk 1y ago
> but getting corrected after tests are run, whereas my agents typically get stuck in these write-and-test loops This maybe a definition problem then. I don’t think “the agent did a dumb thing that it can’t reason out of” is a hallucination. To me a hallucination is a pretty specific failure mode, it invents something that doesn’t exist. Models still do that for me but the build test loop sets them aright on that nearly perfectly. So I guess the model is still hallucinating but the agent isn’t so the output is unimpacted. So I don’t care. For the agent is dumb scenario, I aggressively delete and reprompt. This is something I’ve actually gotten much better at with time and experience, both so it doesn’t happen often and I can course correct quickly. I find it works nearly as well for teaching me about the problem domain as my own mistakes do but is much faster to get to. But if I were going to be pithy. Aggressively deleting work output from an agent is part of their value proposition. They don’t get offended and they don’t need explanations why. Of course they don’t learn well either, that’s on you.