5 ms·

Did you have to do any preparation steps before you asked from a model to do the large scale change or there were no steps involved? For example, did you simply

Did you have to do any preparation steps before you asked from a model to do the large scale change or there were no steps involved? For example, did you simply ask for the change or did you give a model a chance to learn about the codebase. I am genuinely asking, I'm curious because I haven't had a chance to use those models at work.

deleted 2y ago
[deleted]
lolinder 2y ago
Not OP, but I've had the same experience, and that's with tools that purport to handle the context for you. And frankly, if you can't automate context, then you don't have an AI tool that can realistically replace a programmer. If I have to manually select which of my 10000 files are relevant to a given query, then I still need to be in the loop and will also likely end up doing almost as much work as I would have to just write the code.
- menaerus 2y ago
  I see that you deleted your previous response which was unnecessarily snarky while my question was genuine and simple I suppose. > And frankly, if you can't automate context, How about ingesting the whole codebase into the model? I have seen that this is possible with at least one such tool (Devon) and which I believe is using gpt model underneath meaning that other providers could automate this step too. I am curious if that would help in generating more legit large scale changes.
  - lolinder 2y ago
    > I see that you deleted your previous response which was unnecessarily snarky while my question was genuine and simple I suppose. You edited your comment to clarify that you were asking from a place of ignorance as to the tools. Your original comment read as snarky and I responded accordingly, deleting it when I realized that you had changed yours. :) > How about ingesting the whole codebase into the model? I have seen that this is possible with at least one such tool (Devon) and which I believe is using gpt model underneath meaning that other providers could automate this step too. I am curious if that would help in generating more legit large scale changes. It doesn't work. Even the models that claim to have really large context windows get very distracted if you don't selectively pick relevant context. That's why I always talk about useful context window instead of just plain context window—the useful context window is much lower and how much you have depends on the type of text you're feeding it.
    - menaerus 2y ago
      I don't think my comment read as snarky but I was surprised to see the immediate downvote which presumably came from you so I only added the last sentence. This is a stupid way of disagreeing and attempting to shut down the discussion without merits. > It doesn't work. Even the models that claim to have really large context windows get very distracted if you don't selectively pick relevant context. I thought Devon is able to pre-process the whole codebase and which it could take up to a one single day for larger codebases so it must be doing something, e.g. indexing the code? If so, this isn't a context-window specific thing, it's something else and it makes me wonder how that works.
      - lolinder 2y ago
        > I don't think my comment read as snarky but I was surprised to see the immediate downvote which presumably came from you so I only added the last sentence. I can't downvote you because you are downthread of me. HN shadow-disables downvotes on all child and grandchild comments. I'm the one who upvoted you to counteract the downvote. :)
        whamlastxmas 2y ago
        And then they hugged and became lifelong friends :)
        menaerus 2y ago
        You never know - one moment arguing on HN and the second moment you know, drinking at the bar lamenting on how AI is gonna replace us :)
        menaerus 2y ago
        Ok, sorry about that.
  - vunderba 2y ago
    > How about ingesting the whole codebase into the model? You keep referring to this vague idea of "ingesting the whole codebase". What does this even mean? Are you talking about building a code base specific rag, fine tuning against a model, injecting the entire code base into the system context, etc.?
    - menaerus 2y ago
      It is vague because the implementation details you are asking me for are closed source, for obvious reasons. I can only guess what it does but that's besides the point. The point is rather that Devon or 1M window context qwen model might be better or more resilient towards the "lack of context" than what the others were suggesting.
raducu 2y ago
> For example, did you simply ask for the change or did you give a model a chance to learn about the codebase. I've tried it both with personal projects and work. My personal project/benchmark is a 3d snake game. O3 is by far the best, but even with a couple of hundred lines of code it wrote itself it loses coherence and can't produce changes that involve changing 2 lines of code in a span of 50 lines of code. It either cannot comprehend it needs to touch multiple places of re-writes huge chunks of code and breaks other functionality. At work, it's fine for writing unit tests on straight forward tasks that it most likely has seen examples of before. On domain-specific tasks it's not so good and those tasks usually involve multiple file edits in multiple modules. The denser the logic, the smaller the context where LLMs seem to be coherent. And that's funny, because LLMs seem to deal much better with changing code humans wrote than the code the LLMs wrote themselves. Which makes me wonder -- if we're all replaced by AI, who will write the frameworks and programming languages themselves?
- menaerus 2y ago
  Thanks but IIUC you're describing a situation where you're simply using a model without giving it a chance to learn from the whole codebase? If so, then I was asking for the opposite where you would ingest the whole codebase and then let the model spit out the code. This in theory should enable the AI model to build a model of your code. > if we're all replaced by AI, who will write the frameworks and programming languages themselves? What for? There's enough programming languages and there's enough of the frameworks. How about using an AI model to maintain and develop existing complex codebases? IMHO if AI models become more sophisticated and are able to solve this, then the answer is pretty clear who will be doing it.
Jcampuzano2 2y ago
There are simply no models that can keep in context the amount of info required in enterprise codebases before starting to forget or hallucinate. I've tried to give it relevant context myself (a tedious task in itself to be honest) and even tools that claim to automatically be able to do so fail wonderfully at bigger than toy project size in my experience. The codebase I'm working on day to day at this moment is give or take around 800,000 lines of code and this isn't even close to our largest codebase since its just one client app for our monolith. Even trivial changes require touching many files. It would honestly take any average programmer less time to implement something themselves than trying to convince an LLM to complete it.
- menaerus 2y ago
  The largest context that I am aware that an open-source model (e.g. qwen) can manage is 1M tokens. This should translate to ~30kLoC. I'd envision that this could in theory work even on large codebases. It certainly depends on the change to be done but I can imagine that ~30kLoC of context is large enough for most of the module-specific changes. Possibly the models that you're using have a much smaller context window? Then again, and I am repeating myself from other comments I made here in the topic, there's also Devon which pre-processes the codebase before you can do anything else. That kinda makes me wonder if current limitations that people observe in using those tools are really representative of what might be the current state of the art.
  - Jcampuzano2 2y ago
    If you don't mind me asking, what size of codebases do you typically work on? As mentioned I've tried using all the available commercial models and none work better than as a helpful autocomplete, test, and utility function generator. I'm sure maybe big players like Meta, OpenAI, MS, etc do have the capability of expanding its context for their own internal projects and training specifically on their code, but most of the rest of us can't feasibly do that since we don't own our own AI moat. Even on my personal projects and smaller internal projects that are small toy projects or utility tools I sometimes struggle to get them to build anything significant. I'm not saying its impossible, but I always find it best at starting things from scratch, and small tools. Maybe its just a sign that AI would be best for microservices. I've never used Devon so I can't speak to it, but I do recall seeing it was also overhyped at best and struggled to do anything it was purported to be able to in demos. Not saying that this is still true. I would be interested in seeing how Devon performs on a large open source project in real-time (since if I recall their demos were not real-time demonstrations) for instance just to evaluate its capabilities.

deleted 2y ago

[deleted]

lolinder 2y ago

Not OP, but I've had the same experience, and that's with tools that purport to handle the context for you. And frankly, if you can't automate context, then you don't have an AI tool that can realistically replace a programmer. If I have to manually select which of my 10000 files are relevant to a given query, then I still need to be in the loop and will also likely end up doing almost as much work as I would have to just write the code.

menaerus 2y ago
I see that you deleted your previous response which was unnecessarily snarky while my question was genuine and simple I suppose. > And frankly, if you can't automate context, How about ingesting the whole codebase into the model? I have seen that this is possible with at least one such tool (Devon) and which I believe is using gpt model underneath meaning that other providers could automate this step too. I am curious if that would help in generating more legit large scale changes.
- lolinder 2y ago
  > I see that you deleted your previous response which was unnecessarily snarky while my question was genuine and simple I suppose. You edited your comment to clarify that you were asking from a place of ignorance as to the tools. Your original comment read as snarky and I responded accordingly, deleting it when I realized that you had changed yours. :) > How about ingesting the whole codebase into the model? I have seen that this is possible with at least one such tool (Devon) and which I believe is using gpt model underneath meaning that other providers could automate this step too. I am curious if that would help in generating more legit large scale changes. It doesn't work. Even the models that claim to have really large context windows get very distracted if you don't selectively pick relevant context. That's why I always talk about useful context window instead of just plain context window—the useful context window is much lower and how much you have depends on the type of text you're feeding it.
  - menaerus 2y ago
    I don't think my comment read as snarky but I was surprised to see the immediate downvote which presumably came from you so I only added the last sentence. This is a stupid way of disagreeing and attempting to shut down the discussion without merits. > It doesn't work. Even the models that claim to have really large context windows get very distracted if you don't selectively pick relevant context. I thought Devon is able to pre-process the whole codebase and which it could take up to a one single day for larger codebases so it must be doing something, e.g. indexing the code? If so, this isn't a context-window specific thing, it's something else and it makes me wonder how that works.
    - lolinder 2y ago
      > I don't think my comment read as snarky but I was surprised to see the immediate downvote which presumably came from you so I only added the last sentence. I can't downvote you because you are downthread of me. HN shadow-disables downvotes on all child and grandchild comments. I'm the one who upvoted you to counteract the downvote. :)
      - whamlastxmas 2y ago
        And then they hugged and became lifelong friends :)
        menaerus 2y ago
        You never know - one moment arguing on HN and the second moment you know, drinking at the bar lamenting on how AI is gonna replace us :)
      - menaerus 2y ago
        Ok, sorry about that.
- vunderba 2y ago
  > How about ingesting the whole codebase into the model? You keep referring to this vague idea of "ingesting the whole codebase". What does this even mean? Are you talking about building a code base specific rag, fine tuning against a model, injecting the entire code base into the system context, etc.?
  - menaerus 2y ago
    It is vague because the implementation details you are asking me for are closed source, for obvious reasons. I can only guess what it does but that's besides the point. The point is rather that Devon or 1M window context qwen model might be better or more resilient towards the "lack of context" than what the others were suggesting.

raducu 2y ago

> For example, did you simply ask for the change or did you give a model a chance to learn about the codebase. I've tried it both with personal projects and work. My personal project/benchmark is a 3d snake game. O3 is by far the best, but even with a couple of hundred lines of code it wrote itself it loses coherence and can't produce changes that involve changing 2 lines of code in a span of 50 lines of code. It either cannot comprehend it needs to touch multiple places of re-writes huge chunks of code and breaks other functionality. At work, it's fine for writing unit tests on straight forward tasks that it most likely has seen examples of before. On domain-specific tasks it's not so good and those tasks usually involve multiple file edits in multiple modules. The denser the logic, the smaller the context where LLMs seem to be coherent. And that's funny, because LLMs seem to deal much better with changing code humans wrote than the code the LLMs wrote themselves. Which makes me wonder -- if we're all replaced by AI, who will write the frameworks and programming languages themselves?

menaerus 2y ago
Thanks but IIUC you're describing a situation where you're simply using a model without giving it a chance to learn from the whole codebase? If so, then I was asking for the opposite where you would ingest the whole codebase and then let the model spit out the code. This in theory should enable the AI model to build a model of your code. > if we're all replaced by AI, who will write the frameworks and programming languages themselves? What for? There's enough programming languages and there's enough of the frameworks. How about using an AI model to maintain and develop existing complex codebases? IMHO if AI models become more sophisticated and are able to solve this, then the answer is pretty clear who will be doing it.

Jcampuzano2 2y ago

There are simply no models that can keep in context the amount of info required in enterprise codebases before starting to forget or hallucinate. I've tried to give it relevant context myself (a tedious task in itself to be honest) and even tools that claim to automatically be able to do so fail wonderfully at bigger than toy project size in my experience. The codebase I'm working on day to day at this moment is give or take around 800,000 lines of code and this isn't even close to our largest codebase since its just one client app for our monolith. Even trivial changes require touching many files. It would honestly take any average programmer less time to implement something themselves than trying to convince an LLM to complete it.

menaerus 2y ago
The largest context that I am aware that an open-source model (e.g. qwen) can manage is 1M tokens. This should translate to ~30kLoC. I'd envision that this could in theory work even on large codebases. It certainly depends on the change to be done but I can imagine that ~30kLoC of context is large enough for most of the module-specific changes. Possibly the models that you're using have a much smaller context window? Then again, and I am repeating myself from other comments I made here in the topic, there's also Devon which pre-processes the codebase before you can do anything else. That kinda makes me wonder if current limitations that people observe in using those tools are really representative of what might be the current state of the art.
- Jcampuzano2 2y ago
  If you don't mind me asking, what size of codebases do you typically work on? As mentioned I've tried using all the available commercial models and none work better than as a helpful autocomplete, test, and utility function generator. I'm sure maybe big players like Meta, OpenAI, MS, etc do have the capability of expanding its context for their own internal projects and training specifically on their code, but most of the rest of us can't feasibly do that since we don't own our own AI moat. Even on my personal projects and smaller internal projects that are small toy projects or utility tools I sometimes struggle to get them to build anything significant. I'm not saying its impossible, but I always find it best at starting things from scratch, and small tools. Maybe its just a sign that AI would be best for microservices. I've never used Devon so I can't speak to it, but I do recall seeing it was also overhyped at best and struggled to do anything it was purported to be able to in demos. Not saying that this is still true. I would be interested in seeing how Devon performs on a large open source project in real-time (since if I recall their demos were not real-time demonstrations) for instance just to evaluate its capabilities.