4 ms·

Part of the problem is that many working developers are still in companies that don't allow experimentation with the bleeding edge of AI on their code base, so

Part of the problem is that many working developers are still in companies that don't allow experimentation with the bleeding edge of AI on their code base, so their experiences come from headlines and from playing around on personal projects. And on the first 10,000 lines of code, the best in class tools are actually pretty good. Since they can help define the structure of the code, it ends up shaped in a way that works well for the models, and it still basically all fits in the useful context window. What developers who can't use it on large warty codebases don't see is how poorly even the best tools do on the kinds of projects that software engineers typically work on for pay. So they're faced with headlines that oversell AI capabilities and positive experiences with their own small projects and they buy the hype.

throwaway0123_5 2y ago
Some codebases grown with AI assistance must be getting pretty large now, I think an interesting metric to track would be percent of code that is AI generated over time. Still isn't a perfect proxy for how much work the AI is replacing though, because of course it isn't the case that all lines of code would take the same amount of time to write by hand.
- lolinder 2y ago
  Yeah, that would be very helpful to track. Anecdotally, I have found in my own projects that the larger they get the less I can lean on agent/chat models to generate new code that works (without needing enough tweaks that I may as well have just written it myself). Having been written with models does seem to help, but it doesn't get over the problem that eventually you run out of useful context window. What I have seen is that autocomplete scales fine (and Cursor's autocomplete is amazing), but autocomplete supplements a software engineer, it doesn't replace them. So right now I can see a world where one engineer can do a lot more than before, but it's not clear that that will actually reduce engineering jobs in the long term as opposed to just creating a teller effect.
  - ryandrake 2y ago
    It might not just be helpful but required one day. Depending on how the legality around AI-generated code plays out, it's not out of the question that companies using it will have to keep track of and check the provenance and history of their code, like many companies already do for any open source code that may leak into their project. My company has an "open source review" process to help ensure that developers aren't copy-pasting GPL'ed code or including copyleft libraries into our non-GPL licensed products. Perhaps one day it will be common to do an "AI audit" to ensure all code written complied with whatever the future regulatory landscape shapes up to be.
Jcampuzano2 2y ago
My company allowed us to use it but most developers around me didn't reach out to the correct people to be able to use it. Yes I find it incredibly helpful and try to tell them. But it's only helpful in small contexts, auto completing things, small snippets, generating small functions. Any large scale changes like most of these AI companies try to push them being capable of doing it just falls straight on its face. I've tried many times, and with every new model. It can't do it well enough to trust in any codebase that's bigger than a few 10000 lines of code.
- menaerus 2y ago
  Did you have to do any preparation steps before you asked from a model to do the large scale change or there were no steps involved? For example, did you simply ask for the change or did you give a model a chance to learn about the codebase. I am genuinely asking, I'm curious because I haven't had a chance to use those models at work.
  - deleted 2y ago
    [deleted]
  - lolinder 2y ago
    Not OP, but I've had the same experience, and that's with tools that purport to handle the context for you. And frankly, if you can't automate context, then you don't have an AI tool that can realistically replace a programmer. If I have to manually select which of my 10000 files are relevant to a given query, then I still need to be in the loop and will also likely end up doing almost as much work as I would have to just write the code.
    - menaerus 2y ago
      I see that you deleted your previous response which was unnecessarily snarky while my question was genuine and simple I suppose. > And frankly, if you can't automate context, How about ingesting the whole codebase into the model? I have seen that this is possible with at least one such tool (Devon) and which I believe is using gpt model underneath meaning that other providers could automate this step too. I am curious if that would help in generating more legit large scale changes.
WillPostForFood 2y ago
the kinds of projects that software engineers typically work on for pay This assumes a typical project is fairly big and complex. Maybe I'm biased the other way, but I'd guess 90% of software engineers are writing boilerplate code today that could be greatly assisted by LLM tools. E.g., PHP is still one of the top languages, which means a lot of basic WordPress stuff that LLMs are great at.
- lolinder 2y ago
  The question isn't whether the code is complex algorithmically, the question is whether the code is: * Too large to fit in the useful context window of the model, * Filled with bunch of warts and landmines, and * Connected to external systems that are not self-documenting in the code. Most stuff that most of us are working on meets all three of these criteria. Even microservices don't help, if anything they make things worse by pulling the necessary context outside of the code altogether. And note that I'm not saying that the tools aren't useful, I'm saying that they're nowhere near good enough to be threatening to anyone's job.

throwaway0123_5 2y ago

Some codebases grown with AI assistance must be getting pretty large now, I think an interesting metric to track would be percent of code that is AI generated over time. Still isn't a perfect proxy for how much work the AI is replacing though, because of course it isn't the case that all lines of code would take the same amount of time to write by hand.

lolinder 2y ago
Yeah, that would be very helpful to track. Anecdotally, I have found in my own projects that the larger they get the less I can lean on agent/chat models to generate new code that works (without needing enough tweaks that I may as well have just written it myself). Having been written with models does seem to help, but it doesn't get over the problem that eventually you run out of useful context window. What I have seen is that autocomplete scales fine (and Cursor's autocomplete is amazing), but autocomplete supplements a software engineer, it doesn't replace them. So right now I can see a world where one engineer can do a lot more than before, but it's not clear that that will actually reduce engineering jobs in the long term as opposed to just creating a teller effect.
- ryandrake 2y ago
  It might not just be helpful but required one day. Depending on how the legality around AI-generated code plays out, it's not out of the question that companies using it will have to keep track of and check the provenance and history of their code, like many companies already do for any open source code that may leak into their project. My company has an "open source review" process to help ensure that developers aren't copy-pasting GPL'ed code or including copyleft libraries into our non-GPL licensed products. Perhaps one day it will be common to do an "AI audit" to ensure all code written complied with whatever the future regulatory landscape shapes up to be.

Jcampuzano2 2y ago

My company allowed us to use it but most developers around me didn't reach out to the correct people to be able to use it. Yes I find it incredibly helpful and try to tell them. But it's only helpful in small contexts, auto completing things, small snippets, generating small functions. Any large scale changes like most of these AI companies try to push them being capable of doing it just falls straight on its face. I've tried many times, and with every new model. It can't do it well enough to trust in any codebase that's bigger than a few 10000 lines of code.

menaerus 2y ago
Did you have to do any preparation steps before you asked from a model to do the large scale change or there were no steps involved? For example, did you simply ask for the change or did you give a model a chance to learn about the codebase. I am genuinely asking, I'm curious because I haven't had a chance to use those models at work.
- deleted 2y ago
  [deleted]
- lolinder 2y ago
  Not OP, but I've had the same experience, and that's with tools that purport to handle the context for you. And frankly, if you can't automate context, then you don't have an AI tool that can realistically replace a programmer. If I have to manually select which of my 10000 files are relevant to a given query, then I still need to be in the loop and will also likely end up doing almost as much work as I would have to just write the code.
  - menaerus 2y ago
    I see that you deleted your previous response which was unnecessarily snarky while my question was genuine and simple I suppose. > And frankly, if you can't automate context, How about ingesting the whole codebase into the model? I have seen that this is possible with at least one such tool (Devon) and which I believe is using gpt model underneath meaning that other providers could automate this step too. I am curious if that would help in generating more legit large scale changes.

WillPostForFood 2y ago

the kinds of projects that software engineers typically work on for pay This assumes a typical project is fairly big and complex. Maybe I'm biased the other way, but I'd guess 90% of software engineers are writing boilerplate code today that could be greatly assisted by LLM tools. E.g., PHP is still one of the top languages, which means a lot of basic WordPress stuff that LLMs are great at.

lolinder 2y ago
The question isn't whether the code is complex algorithmically, the question is whether the code is: * Too large to fit in the useful context window of the model, * Filled with bunch of warts and landmines, and * Connected to external systems that are not self-documenting in the code. Most stuff that most of us are working on meets all three of these criteria. Even microservices don't help, if anything they make things worse by pulling the necessary context outside of the code altogether. And note that I'm not saying that the tools aren't useful, I'm saying that they're nowhere near good enough to be threatening to anyone's job.