4 ms·

Based on my experiences having spent a bunch of time experimenting with LLMs for writing code, I think the hardest part that I haven't yet seen a solution to is

Based on my experiences having spent a bunch of time experimenting with LLMs for writing code, I think the hardest part that I haven't yet seen a solution to is modifying existing code bases. Sure, if you're doing greenfield, just ask it to write a new file. If you only have a simple script, you can ask it to rewrite the whole file. The tricky bit however, is figuring out how to edit relevant parts of the codebase, keeping in mind that the context window of LLMs is very limited. You basically want it to navigate to the right part of the codebase, then scope it to just part of a file that is relevant, and then let it rewrite just that small bit. I'm not saying it's impossible (maybe with proper indexing with embeddings, and splitting files up by i.e. functions you can make it work), but I think it's very non-trivial. Anyway, good luck! I hope you'll share your learnings once you figure this out. I think the idea of putting LLMs into Firecracker VMs to contain them is a very cool approach.

chatmasta 4y ago
I think this will get easier as the supported context sizes get larger. Of course the tooling needs to take care of most of the drudgery, but I'm not sure there's any underlying limitation of an LLM that makes refactoring existing code any different from creating new code. It's just a matter of feeding it the layout of the repo and the code itself in a way that the LLM can retrieve and focus on the relevant files when you ask it for help with a task.
- cube2222 4y ago
  Oh yeah, I agree with the refactoring bit. As I said, rewriting a file (section) works great if your file fits into the context window. But context windows are far from being large enough to fit entire repos, nor even entire files (if they're big). I'm not sure how hard just scaling up the context window is, from the current early access of OpenAI GPT-4. > in a way that the LLM can retrieve and focus on the relevant files This I think is something we haven't really figured out yet, esp. if a feature requires working on multiple files. I wouldn't be surprised if approaches based on the semantic level (actually understanding the code and the relationships between it's parts; not the textual representation of it) won't be needed in the end here.
  - chatmasta 4y ago
    I agree retrieval will need to be aligned with semantic layout of the codebase. But that should be pretty straight forward, given the number of static analysis and refactoring tools we already have available to us and that we use daily as part of our IDE workflows. This also implies that the first codebases to really benefit from LLM collaboration will be those written in strongly typed languages which are already amenable to static analysis. And in terms of context windows, it's not like humans keep the entire codebase in their head at all times either. As a developer, when I'm focused on a single task, I'm only ever switching between a handful of files. And that's by design; we build our codebases using abstractions that are understandable to humans, given our limited context window, with its well-known limit of about seven simultaneous registers. So if anything, perhaps the risk of introducing an LLM to a codebase is that it could create abstractions that are more complicated than what a human would prefer to read and maintain.
M4v3R 4y ago
This is actually possible, I made a proof of concept agent that does exactly this. The agent has access to several commands that get executed on demand - ListFiles, ReadFile and WriteFile. You give it a prompt - for example "add a heading to the homepage and make it yellow" and it will use ListFiles to locate the correct file, then use ReadFile to read it and then finally use WriteFile to write the new contents of that file. It was just a PoC so it wasn't bulletproof and I only ever made it work on a small Next.js codebase (with every file being intentionally small so it fits the context window) but it did work correctly.

chatmasta 4y ago

I think this will get easier as the supported context sizes get larger. Of course the tooling needs to take care of most of the drudgery, but I'm not sure there's any underlying limitation of an LLM that makes refactoring existing code any different from creating new code. It's just a matter of feeding it the layout of the repo and the code itself in a way that the LLM can retrieve and focus on the relevant files when you ask it for help with a task.

cube2222 4y ago
Oh yeah, I agree with the refactoring bit. As I said, rewriting a file (section) works great if your file fits into the context window. But context windows are far from being large enough to fit entire repos, nor even entire files (if they're big). I'm not sure how hard just scaling up the context window is, from the current early access of OpenAI GPT-4. > in a way that the LLM can retrieve and focus on the relevant files This I think is something we haven't really figured out yet, esp. if a feature requires working on multiple files. I wouldn't be surprised if approaches based on the semantic level (actually understanding the code and the relationships between it's parts; not the textual representation of it) won't be needed in the end here.
- chatmasta 4y ago
  I agree retrieval will need to be aligned with semantic layout of the codebase. But that should be pretty straight forward, given the number of static analysis and refactoring tools we already have available to us and that we use daily as part of our IDE workflows. This also implies that the first codebases to really benefit from LLM collaboration will be those written in strongly typed languages which are already amenable to static analysis. And in terms of context windows, it's not like humans keep the entire codebase in their head at all times either. As a developer, when I'm focused on a single task, I'm only ever switching between a handful of files. And that's by design; we build our codebases using abstractions that are understandable to humans, given our limited context window, with its well-known limit of about seven simultaneous registers. So if anything, perhaps the risk of introducing an LLM to a codebase is that it could create abstractions that are more complicated than what a human would prefer to read and maintain.

M4v3R 4y ago

This is actually possible, I made a proof of concept agent that does exactly this. The agent has access to several commands that get executed on demand - ListFiles, ReadFile and WriteFile. You give it a prompt - for example "add a heading to the homepage and make it yellow" and it will use ListFiles to locate the correct file, then use ReadFile to read it and then finally use WriteFile to write the new contents of that file. It was just a PoC so it wasn't bulletproof and I only ever made it work on a small Next.js codebase (with every file being intentionally small so it fits the context window) but it did work correctly.