12 ms·

I'm 100% an AI skeptic. I don't think we're near AGI, well any reasonable definition of it. I don't think we're remotely near a super intelligence or the singul

I'm 100% an AI skeptic. I don't think we're near AGI, well any reasonable definition of it. I don't think we're remotely near a super intelligence or the singularity or whatever. But that doesn't matter. These tools will change how work is done. What matters is what he says here: > Peeps, let’s do some really simple back-of-envelope math. Trust me, it won’t be difficult math. > You get the LLM to draft some code for you that’s 80% complete/correct. > You tweak the last 20% by hand. > How much of a productivity increase is that? Well jeepers, if you’re only doing 1/5th the work, then you are… punches buttons on calculator watch… five times as productive. The hard part is the engineering, yes, and now we can actually focus on it. 90% of the work that software engineers do isn't novel. People aren't sitting around coding complicated algorithms that require PhDs. Most of the work is doing pretty boring stuff. And even if LLMs can only write 50% of that you're still going to get a massive boost in productivity. Of course this isn't always going to be great. Debugging this code is probably going to suck, and even with the best engineering it'll likely accelerate big balls of mud. But it's going to enable a lot of code to get written by a lot more people and it's super exciting.

eric-hu 2y ago
> I don't think we're near AGI, well any reasonable definition of it. But wait just hear me out for a second! What if we've also sufficiently lowered the definition of "GI" at the same time...
yoyohello13 2y ago
> But it's going to enable a lot of code to get written by a lot more people and it's super exciting. Is that exciting though? A lot of the code we've currently got is garbage. We are already slogging through a morass of code. I could be wrong, but I don't think LLMs are going to change this trend.
- trescenzi 2y ago
  I think it's exciting that more people will be able to build more things. I agree though that most code is garbage, and that it's likely going to get worse. The future can be both concerning and exciting.
- bdangubic 2y ago
  it is GREAT time to slowly get into the consulting side of our industry. there will be so much mess that I think there will be troves of opportunity to come in on a contract and fix messes. there always was but there will be a ton more!
- hirvi74 2y ago
  > A lot of the code we've currently got is garbage. That is why I have never understood the harsh criticism of quality of LLM generated code. What do people think much of the models were trained on? Garbage in -> Garbage out.
deadbabe 2y ago
The original calculation is not even right, because 100% of the work software engineers do probably isn’t even writing code. A lot of the time they’re debugging code or attending some kind of meeting. So if only 20% of their time is coding and they do 1/5th of it, they’re like what, 16% more productive overall? Now factor in the expense of using LLMs, it’s not worth it, the math isn’t mathing. You can’t even squeeze a 2x improvement, let alone the 5-10x that has become expected in the tech industry.
- JohnMakin 2y ago
  Also absent in these discussions is who maintains the 80% that wasn’t written by them when it needs to either be fixed or adapted to a new use case. This is where I typically see “fast” AI prototypes fall over completely. Will that get better? I’m ambivalent but could concede ‘probably’ but at the end of the day I don’t think you can get away from the fact that for every non trivial programming task, there does need to be a human somewhere to verify/validate work, or at the very least understand it enough to debug it. This seems like an inescapable fact.
  - deadbabe 2y ago
    And one thing people don’t realize is that if you’re working in a codebase heavily coded by AI, you’re going to want to see the original prompts used if you don’t know the original engineers intent. And no one seems to track this. It becomes a game of Chinese whispers. Honestly this quickly becomes a shit show and I don’t know how anyone can seriously consider having a large amount of a codebase coded by AI.
    - trescenzi 2y ago
      How is this different from any legacy code base? Everywhere I've worked there's been swaths of code where the initial intent is lost to time and debugging them is basically archeology. Random AI code isn't going to be categorically different from random code copied from StackOverflow, or code simply written a few years ago. Reading code will always be harder than writing it.
Terr_ 2y ago
My concern is that it's always harder to comprehend and debug code you didn't write, so there are hidden costs to using a copy-pasted solution even if you "get to the testing phase faster." Shaving percentages in one place could add different ones elsewhere. Imagine if your manager announced a wonderful new productivity process: From now on a herd of excited interns will get the first stab at feature requests, and allll that the senior developers need to do is take that output and juuuust spruce it up a bit. Then you have plenty of time for those TPS reports. (Now, later in my career, I have a lot more sympathy for Tom Smykowski, speaking with customers so the engineers don't have to.)
accrual 2y ago
Personal anecdote: I pushed some LLM generated code into Prod during a call with the client today. It was simple stuff, just a string manipulation function, about 10 LOC. Absolutely could have done it by hand but it was really cool to not have to think about the details - I just gave some inputs and desired outputs and had the function in hand and tested it. Instead of saying "I'll reply when the code is done" I was able to say "it's done, go ahead and continue" and that was a cool moment for me.
- roland35 2y ago
  That's the perfect application for llm code! It is a well defined problem, with known inputs and outputs, and probably many examples the llm slurped up from open source code. And that isn't a bad thing! Now us engineers need to focus on the high level stuff, like what code to write in the first place.
stouset 2y ago
> Of course this isn't always going to be great. Debugging this code is probably going to suck, and even with the best engineering it'll likely accelerate big balls of mud. But it's going to enable a lot of code to get written by a lot more people and it's super exciting. This is quite literally what junior engineers are. I've worked at many places filled to the brim with junior engineers. They weren't enjoyable places to work. Those businesses did not succeed. They sure did produce a lot of code though.
- EA-3167 2y ago
  I'd also be very suspicious of this Pareto distribution, to me it implies that the 20% of work we're talking about is the hard part. Spitting out code without understanding it definitely sounds like the easy part, the part that doesn't require much if any thinking. I'd be much more interested to see a breakdown of TIME, not volume of code; how much time does using an LLM save (or not) in the context of a given task?
  - onemoresoop 2y ago
    Whatever time is be saved (if?) will be taken up by the higher expectation of the worker. Imagine a manager’s monologue “now that you have AI, wouldnt’t it be easier to implement those features that we had to scrap because we were understaffed? Soon we’ll be even more understaffed and you’ll have to work harder.” Let’s see hope my pessimism is unfounded.
- seba_dos1 2y ago
  The thing with junior engineers is that they may eventually become proper seniors and you'll play your role in it by working with them. Nobody learns anything from working with LLMs. You can't become a senior and skip being a junior.
pton_xd 2y ago
> How much of a productivity increase is that? Well jeepers, if you’re only doing 1/5th the work, then you are… punches buttons on calculator watch… five times as productive. To state the obvious -- only if all units of work are of equal difficulty.
- NoGravitas 2y ago
  The last 20% always takes 80% of the effort, of course.
hirvi74 2y ago
> But it's going to enable a lot of code to get written by a lot more people and it's super exciting. I think it will cut both ways. I guess it depends on the type of work. I feel like expectations will fill to meet the space of the increased productivity. At least, I can see this being true in a professional sense. If I could be five times more productive i.e., work 80% less, then I would be quite excited. However, I imagine most of us will be still be working just as much -- if not more. Maybe I am just jaded, but I can't help but be reminded of the old saying, "The reward for digging the biggest hole is a bigger shovel." For personal endeavors, I am nothing but excited for this technology.
- thunky 2y ago
  > I imagine most of us will be still be working just as much -- if not more Yes but there will be far less of us working. Which, in a healthy society would actually be a good thing.
  - hirvi74 2y ago
    > Which, in a healthy society would actually be a good thing. The bad news is that we do not and have not ever lived in one.
j45 2y ago
This isn't about the promise of AI to me, it's more about this type of AI technology, which should more accurately be called LLMs, as there are others. Skepticism aside, looking at what's been possible or similar, much like Microsoft word introduces more and more grammar related features, LLMs are reasonably decent for the use case of supplying your input to it and speeding up iterations with you at the drivers seat. Whether it's text in a word processor, or code in an editor, there is likely some meaningful improvements to not dismiss the technology. The inverse of that, where someone wants a magic seed prompt to do everything, could prove to be a bit more challenging where existing models are trained on what they know, and steps to move beyond that where they learn appear to be a few years away.

eric-hu 2y ago

> I don't think we're near AGI, well any reasonable definition of it. But wait just hear me out for a second! What if we've also sufficiently lowered the definition of "GI" at the same time...

yoyohello13 2y ago

> But it's going to enable a lot of code to get written by a lot more people and it's super exciting. Is that exciting though? A lot of the code we've currently got is garbage. We are already slogging through a morass of code. I could be wrong, but I don't think LLMs are going to change this trend.

trescenzi 2y ago
I think it's exciting that more people will be able to build more things. I agree though that most code is garbage, and that it's likely going to get worse. The future can be both concerning and exciting.
bdangubic 2y ago
it is GREAT time to slowly get into the consulting side of our industry. there will be so much mess that I think there will be troves of opportunity to come in on a contract and fix messes. there always was but there will be a ton more!
hirvi74 2y ago
> A lot of the code we've currently got is garbage. That is why I have never understood the harsh criticism of quality of LLM generated code. What do people think much of the models were trained on? Garbage in -> Garbage out.

deadbabe 2y ago

The original calculation is not even right, because 100% of the work software engineers do probably isn’t even writing code. A lot of the time they’re debugging code or attending some kind of meeting. So if only 20% of their time is coding and they do 1/5th of it, they’re like what, 16% more productive overall? Now factor in the expense of using LLMs, it’s not worth it, the math isn’t mathing. You can’t even squeeze a 2x improvement, let alone the 5-10x that has become expected in the tech industry.

JohnMakin 2y ago
Also absent in these discussions is who maintains the 80% that wasn’t written by them when it needs to either be fixed or adapted to a new use case. This is where I typically see “fast” AI prototypes fall over completely. Will that get better? I’m ambivalent but could concede ‘probably’ but at the end of the day I don’t think you can get away from the fact that for every non trivial programming task, there does need to be a human somewhere to verify/validate work, or at the very least understand it enough to debug it. This seems like an inescapable fact.
- deadbabe 2y ago
  And one thing people don’t realize is that if you’re working in a codebase heavily coded by AI, you’re going to want to see the original prompts used if you don’t know the original engineers intent. And no one seems to track this. It becomes a game of Chinese whispers. Honestly this quickly becomes a shit show and I don’t know how anyone can seriously consider having a large amount of a codebase coded by AI.
  - trescenzi 2y ago
    How is this different from any legacy code base? Everywhere I've worked there's been swaths of code where the initial intent is lost to time and debugging them is basically archeology. Random AI code isn't going to be categorically different from random code copied from StackOverflow, or code simply written a few years ago. Reading code will always be harder than writing it.

Terr_ 2y ago

My concern is that it's always harder to comprehend and debug code you didn't write, so there are hidden costs to using a copy-pasted solution even if you "get to the testing phase faster." Shaving percentages in one place could add different ones elsewhere. Imagine if your manager announced a wonderful new productivity process: From now on a herd of excited interns will get the first stab at feature requests, and allll that the senior developers need to do is take that output and juuuust spruce it up a bit. Then you have plenty of time for those TPS reports. (Now, later in my career, I have a lot more sympathy for Tom Smykowski, speaking with customers so the engineers don't have to.)

accrual 2y ago

Personal anecdote: I pushed some LLM generated code into Prod during a call with the client today. It was simple stuff, just a string manipulation function, about 10 LOC. Absolutely could have done it by hand but it was really cool to not have to think about the details - I just gave some inputs and desired outputs and had the function in hand and tested it. Instead of saying "I'll reply when the code is done" I was able to say "it's done, go ahead and continue" and that was a cool moment for me.

roland35 2y ago
That's the perfect application for llm code! It is a well defined problem, with known inputs and outputs, and probably many examples the llm slurped up from open source code. And that isn't a bad thing! Now us engineers need to focus on the high level stuff, like what code to write in the first place.

stouset 2y ago

> Of course this isn't always going to be great. Debugging this code is probably going to suck, and even with the best engineering it'll likely accelerate big balls of mud. But it's going to enable a lot of code to get written by a lot more people and it's super exciting. This is quite literally what junior engineers are. I've worked at many places filled to the brim with junior engineers. They weren't enjoyable places to work. Those businesses did not succeed. They sure did produce a lot of code though.

EA-3167 2y ago
I'd also be very suspicious of this Pareto distribution, to me it implies that the 20% of work we're talking about is the hard part. Spitting out code without understanding it definitely sounds like the easy part, the part that doesn't require much if any thinking. I'd be much more interested to see a breakdown of TIME, not volume of code; how much time does using an LLM save (or not) in the context of a given task?
- onemoresoop 2y ago
  Whatever time is be saved (if?) will be taken up by the higher expectation of the worker. Imagine a manager’s monologue “now that you have AI, wouldnt’t it be easier to implement those features that we had to scrap because we were understaffed? Soon we’ll be even more understaffed and you’ll have to work harder.” Let’s see hope my pessimism is unfounded.
seba_dos1 2y ago
The thing with junior engineers is that they may eventually become proper seniors and you'll play your role in it by working with them. Nobody learns anything from working with LLMs. You can't become a senior and skip being a junior.

pton_xd 2y ago

> How much of a productivity increase is that? Well jeepers, if you’re only doing 1/5th the work, then you are… punches buttons on calculator watch… five times as productive. To state the obvious -- only if all units of work are of equal difficulty.

NoGravitas 2y ago
The last 20% always takes 80% of the effort, of course.

hirvi74 2y ago

> But it's going to enable a lot of code to get written by a lot more people and it's super exciting. I think it will cut both ways. I guess it depends on the type of work. I feel like expectations will fill to meet the space of the increased productivity. At least, I can see this being true in a professional sense. If I could be five times more productive i.e., work 80% less, then I would be quite excited. However, I imagine most of us will be still be working just as much -- if not more. Maybe I am just jaded, but I can't help but be reminded of the old saying, "The reward for digging the biggest hole is a bigger shovel." For personal endeavors, I am nothing but excited for this technology.

thunky 2y ago
> I imagine most of us will be still be working just as much -- if not more Yes but there will be far less of us working. Which, in a healthy society would actually be a good thing.
- hirvi74 2y ago
  > Which, in a healthy society would actually be a good thing. The bad news is that we do not and have not ever lived in one.

j45 2y ago

This isn't about the promise of AI to me, it's more about this type of AI technology, which should more accurately be called LLMs, as there are others. Skepticism aside, looking at what's been possible or similar, much like Microsoft word introduces more and more grammar related features, LLMs are reasonably decent for the use case of supplying your input to it and speeding up iterations with you at the drivers seat. Whether it's text in a word processor, or code in an editor, there is likely some meaningful improvements to not dismiss the technology. The inverse of that, where someone wants a magic seed prompt to do everything, could prove to be a bit more challenging where existing models are trained on what they know, and steps to move beyond that where they learn appear to be a few years away.