7 ms·

Have you done much work on using GPT to edit code in an existing codebase? That's been my focus lately, working on my open source GPT coding tool [0]. Gener

Have you done much work on using GPT to *edit* code in an existing codebase? That's been my focus lately, working on my open source GPT coding tool [0]. Generating new code from whole-cloth seems like an easier task for GPT. My tool can certainly do that, as can smol-developer, etc. But you really only do that "once" per project. Can folks use gpt-engineer to modify and extend the code it has already created, as the user comes up with new features, etc? Can it be used to work on a pre-existing codebase? [0] https://github.com/paul-gauthier/aider https://github.com/paul-gauthier/aider

anton-107 3y ago
Hmm, if generating new code is an easy task for GPT, why don't you ask it to create a new project from scratch every moment a user comes up with a new feature? Who needs to maintain an old codebase if you can rewrite it adding new features at whim?
antonoo 3y ago
Not yet. Considered adding it soon, the only reason I decided against — for now — is that for automatic evaluation human edits make it be try difficult! Fully automatic can be evaluated fully automatically. Good input.
headcanon 3y ago
Looks like you know more about it than me, but it seems to me that the main challenge is being able to include the appropriate context, and train it to output a diff which you can then apply to the codebase. Definitely looking forward to the day I review my first AI-generated PR (beyond dependency updates of course).
- anotherpaulg 3y ago
  Yup, those seem to be the key challenges. I've been making good progress on them, but there's plenty more work to do! On the topic of "AI-generated PRs", I used my tool to file a PR to the `glow` CLI tool. I don't know the go language, so I had my tool `aider` add the feature I needed. I mostly use glow to preview README.md for GitHub, so I wanted it to render line breaks like GitHub does. https://github.com/charmbracelet/glow/pull/502 https://github.com/charmbracelet/glow/pull/502 I've also been able solve a couple of github issues that were file by users by just pasting the issue into my tool... it fixed itself. Links below: https://github.com/paul-gauthier/aider/issues/13#issuecomment-1575564422 https://github.com/paul-gauthier/aider/issues/13#issuecommen... https://github.com/paul-gauthier/aider/issues/5#issuecomment-1555021491 https://github.com/paul-gauthier/aider/issues/5#issuecomment...
  - jonrouach 3y ago
    Love this! I've been iterating on a similar project, Talk-to-Repo. It uses retrieval-augmented-generation to access the relevant parts of the code, and lets you chat and collect which code pieces you want. Got stuck at generating good diffs, I'll be sure to look at how you've done it! btw i started my project by turning another project, "Twitter explainer", on itself. It loaded its own code, i asked it to add new features and copy-pasted the results (with some tweaking and occasional trips to phind.com )... :) https://github.com/Arjeo-Inc/talk-to-repo https://github.com/Arjeo-Inc/talk-to-repo original project by Mark Tennenholtz: https://twitter.com/marktenenholtz/status/1651568107192983553 https://twitter.com/marktenenholtz/status/165156810719298355...
- isaacfung 3y ago
  Does anyone know if models like wizardcoder are trained on finished code only or they have been trained on ticket/PR/commit messages and the diff (with the interface of related code provided as context)?
ghughes 3y ago
Cool project! I'm working in the same space (a ChatGPT plugin that can edit files within a shared VS Code workspace) and have built something similar to your "repo map" concept, except slightly lower-level: what you might call a "file map" generated by selectively collapsing AST nodes to fit within the available token budget. If ctags isn't cutting it for you, have a look at tree-sitter [1]. It can generate ASTs for most languages and has a nice API. [1] https://tree-sitter.github.io/tree-sitter/ https://tree-sitter.github.io/tree-sitter/
- anotherpaulg 3y ago
  Glad to hear there are others working on similar things. I've been wishing there was a good forum for like minded folks to share ideas about AI coding, beyond the random drive-by commenting that happens here on HN. I have been looking at tree-sitter quite a bit actually. I love that it has broad language support, which is a key design goal for my tool. My only hesitation is that it doesn't appear to correctly identify multi-line function signatures & calls. If you look below at create, io.tool_error and __init__ you can see that the (row,col)-(row,col) indicies only reference the first line. GPT would really benefit from seeing the entire function signature and call sites. $ tree-sitter tags aider/coders/base_coder.py ... create | function def (39, 8) - (39, 14) `def create(` check_model_availability | call ref (54, 19) - (54, 43) `if not check_model_availability(main_model):` tool_error | call ref (56, 23) - (56, 33) `io.tool_error(` EditBlockCoder | call ref (66, 19) - (66, 33) `return EditBlockCoder(main_model, io, **kwargs)` ... __init__ | function def (74, 8) - (74, 16) `def __init__(` set | call ref (89, 26) - (89, 29) `self.abs_fnames = set()` ...
  - deathmonger5000 3y ago
    I too would be interested in a good forum for like minded folks to share ideas about AI coding!
    - IanCal 3y ago
      Same. I don't know if it'd be Lemmy or something else just hosted somewhere, and there might just be 5 of us - if people are interested I can make something today. Strong preferences on platforms? If not I'll make something.
andsoitis 3y ago
> Can it be used to work on a pre-existing codebase? For that, I posit the system would need to understand the existing code base. Not just what the code does, but the intent and the why. I'll leave it up to the reader to decide whether they believe LLMs understand anything. I know where I stand.
- grugagag 3y ago
  Im there with you, LLMs don’t understand in the same sense we do. But they can transform a well written compressed spec into code. The spec becomes the true source and can be tweaked on and chatgpt regenerates the whole code over. It’s nondeterministic so everytime it will be slightly different but with multiple renditions its average should look like the travel salesman graph of the problem itself
jrockway 3y ago
This is something that I've also been wanting to play with. The token limit is too small for my codebase so I haven't really bothered, but it would be nice to tell it all my code and then say "refactor everything of the form `err = foo` into `if err := foo; ...`". If AI is going to take my job, it is going to have to learn to do maintenance!
- anotherpaulg 3y ago
  Give aider a try. As long as each file fits in the context window, it should work. With gpt-3.5-turbo-16k or gpt-4 you can edit files up to around 30 kbytes in size. If you have access to gpt-4-32k, you could edit files up to about 120 kbytes. See the notes here for more info: https://github.com/paul-gauthier/aider#gpt-4-vs-gpt-35 https://github.com/paul-gauthier/aider#gpt-4-vs-gpt-35 I use GPT for all kinds of busy work, and code quality type work. Adding test cases or quality of life features. Things that I might not have the energy to do myself. GPT can often accomplish these tasks off a 1-2 sentence request. Or if not, it will do all the boilerplate and get you 80% of the way there and it's easy to polish up the final 20%.
- NicoJuicy 3y ago
  Perhaps it helps: Don't entirely copy-paste your code base. Eg. I adjusted the prompt to the logical separation in my project ( Eg. the module and the task of the class can be derived from it's namespace) and then indexed the codebase through the namespace + classes + method names with their parameters. It helps adding more codebase to the prompt.
rane 3y ago
How have you created the svg screencast in the README?
- andrewescott 3y ago
  Looking at the source, it appears to have been generated by https://github.com/nbedos/termtosvg https://github.com/nbedos/termtosvg
  - anotherpaulg 3y ago
    Yup, that's what I used. It produces really nice, crisp SVG screencasts.
braindead_in 3y ago
I've played around with aider trying to run tests and fix the code, but it just crashes after exceeding the context window. I am now trying to repurpose the AutoGPT example in langchain.
- anotherpaulg 3y ago
  Sounds like maybe your source files are bigger than the context window? Try including fewer files in the chat. See some notes on GPT models and file sizes here: https://github.com/paul-gauthier/aider#gpt-4-vs-gpt-35 https://github.com/paul-gauthier/aider#gpt-4-vs-gpt-35 I also have some incoming improvements so the tool is more graceful and helpful if you hit the context window limit. And in general, most of my main efforts are focused on making it possible to work with larger and larger codebases in spite of the context window limitations. Please do file an issue with more details on your problems. I can try and help and give you updates if I make improvements to the tool which could solve your use case.
lgas 3y ago
It's interesting to me that you say > Generating new code from whole-cloth seems like an easier task for GPT. as I've been working on a similar tool of my own and I've found the opposite. The initial task I've been trying to have it complete is to build something like https://craigmbooth.com/projects/killer-sudoku-calculator/ https://craigmbooth.com/projects/killer-sudoku-calculator/ but with my bespoke requirements. If I try to specify it up front and have it generate the whole thing it almost always fails in a number of ways at once, despite the requirements being relatively straightforward and clear. However I have had success by walking it through the process step by step along the lines of "please create an index.html that references react from a cdn. It should draw a blank sudoku board" -> "Please add a button that says 'add a cage'. When the button is clicked a box labeled "Cage 1", "Cage 2", etc should be added to a list to the right of the board" -> "Please track the currently selected cage and set the background of the corresponding cage box to green", etc. Likewise, I added the initial set of functions it could access ("Create a File", "Update a File", "Remove a File") manually, but then I had it add additional commands ("Add a directory", "Remove a directory", "Copy a file", etc) and it was able to do it correctly on the first try each time because the pattern already existed.
- anotherpaulg 3y ago
  I think we agree, but maybe I wasn't writing clearly. You're describing building a green field app starting from nothing, step by step. GPT shines at things like this, because they are by definition small code bases. You can probably fit the whole codebase into the context window. Also, your approach of walking it through step-by-step is perfect, since you get to guide it to build a wise code architecture as you go. It's hard to naively point GPT at a big, existing repo and try and do non-trivial changes to that codebase. Without a bunch of tooling to help it understand the overall codebase, it won't understand or respect the existing modules, abstractions, etc. It will just start trying to write code in a vacuum, which probably isn't the right thing to do when modifying an existing codebase.

Have you done much work on using GPT to *edit* code in an existing codebase? That's been my focus lately, working on my open source GPT coding tool [0]. Gener

anton-107 3y ago

Hmm, if generating new code is an easy task for GPT, why don't you ask it to create a new project from scratch every moment a user comes up with a new feature? Who needs to maintain an old codebase if you can rewrite it adding new features at whim?

antonoo 3y ago

Not yet. Considered adding it soon, the only reason I decided against — for now — is that for automatic evaluation human edits make it be try difficult! Fully automatic can be evaluated fully automatically. Good input.

headcanon 3y ago

Looks like you know more about it than me, but it seems to me that the main challenge is being able to include the appropriate context, and train it to output a diff which you can then apply to the codebase. Definitely looking forward to the day I review my first AI-generated PR (beyond dependency updates of course).

anotherpaulg 3y ago
Yup, those seem to be the key challenges. I've been making good progress on them, but there's plenty more work to do! On the topic of "AI-generated PRs", I used my tool to file a PR to the `glow` CLI tool. I don't know the go language, so I had my tool `aider` add the feature I needed. I mostly use glow to preview README.md for GitHub, so I wanted it to render line breaks like GitHub does. https://github.com/charmbracelet/glow/pull/502 https://github.com/charmbracelet/glow/pull/502 I've also been able solve a couple of github issues that were file by users by just pasting the issue into my tool... it fixed itself. Links below: https://github.com/paul-gauthier/aider/issues/13#issuecomment-1575564422 https://github.com/paul-gauthier/aider/issues/13#issuecommen... https://github.com/paul-gauthier/aider/issues/5#issuecomment-1555021491 https://github.com/paul-gauthier/aider/issues/5#issuecomment...
- jonrouach 3y ago
  Love this! I've been iterating on a similar project, Talk-to-Repo. It uses retrieval-augmented-generation to access the relevant parts of the code, and lets you chat and collect which code pieces you want. Got stuck at generating good diffs, I'll be sure to look at how you've done it! btw i started my project by turning another project, "Twitter explainer", on itself. It loaded its own code, i asked it to add new features and copy-pasted the results (with some tweaking and occasional trips to phind.com )... :) https://github.com/Arjeo-Inc/talk-to-repo https://github.com/Arjeo-Inc/talk-to-repo original project by Mark Tennenholtz: https://twitter.com/marktenenholtz/status/1651568107192983553 https://twitter.com/marktenenholtz/status/165156810719298355...
isaacfung 3y ago
Does anyone know if models like wizardcoder are trained on finished code only or they have been trained on ticket/PR/commit messages and the diff (with the interface of related code provided as context)?

ghughes 3y ago

Cool project! I'm working in the same space (a ChatGPT plugin that can edit files within a shared VS Code workspace) and have built something similar to your "repo map" concept, except slightly lower-level: what you might call a "file map" generated by selectively collapsing AST nodes to fit within the available token budget. If ctags isn't cutting it for you, have a look at tree-sitter [1]. It can generate ASTs for most languages and has a nice API. [1] https://tree-sitter.github.io/tree-sitter/ https://tree-sitter.github.io/tree-sitter/

anotherpaulg 3y ago
Glad to hear there are others working on similar things. I've been wishing there was a good forum for like minded folks to share ideas about AI coding, beyond the random drive-by commenting that happens here on HN. I have been looking at tree-sitter quite a bit actually. I love that it has broad language support, which is a key design goal for my tool. My only hesitation is that it doesn't appear to correctly identify multi-line function signatures & calls. If you look below at create, io.tool_error and __init__ you can see that the (row,col)-(row,col) indicies only reference the first line. GPT would really benefit from seeing the entire function signature and call sites. $ tree-sitter tags aider/coders/base_coder.py ... create | function def (39, 8) - (39, 14) `def create(` check_model_availability | call ref (54, 19) - (54, 43) `if not check_model_availability(main_model):` tool_error | call ref (56, 23) - (56, 33) `io.tool_error(` EditBlockCoder | call ref (66, 19) - (66, 33) `return EditBlockCoder(main_model, io, **kwargs)` ... __init__ | function def (74, 8) - (74, 16) `def __init__(` set | call ref (89, 26) - (89, 29) `self.abs_fnames = set()` ...
- deathmonger5000 3y ago
  I too would be interested in a good forum for like minded folks to share ideas about AI coding!
  - IanCal 3y ago
    Same. I don't know if it'd be Lemmy or something else just hosted somewhere, and there might just be 5 of us - if people are interested I can make something today. Strong preferences on platforms? If not I'll make something.

andsoitis 3y ago

> Can it be used to work on a pre-existing codebase? For that, I posit the system would need to understand the existing code base. Not just what the code does, but the intent and the why. I'll leave it up to the reader to decide whether they believe LLMs understand anything. I know where I stand.

grugagag 3y ago
Im there with you, LLMs don’t understand in the same sense we do. But they can transform a well written compressed spec into code. The spec becomes the true source and can be tweaked on and chatgpt regenerates the whole code over. It’s nondeterministic so everytime it will be slightly different but with multiple renditions its average should look like the travel salesman graph of the problem itself

jrockway 3y ago

This is something that I've also been wanting to play with. The token limit is too small for my codebase so I haven't really bothered, but it would be nice to tell it all my code and then say "refactor everything of the form `err = foo` into `if err := foo; ...`". If AI is going to take my job, it is going to have to learn to do maintenance!

anotherpaulg 3y ago
Give aider a try. As long as each file fits in the context window, it should work. With gpt-3.5-turbo-16k or gpt-4 you can edit files up to around 30 kbytes in size. If you have access to gpt-4-32k, you could edit files up to about 120 kbytes. See the notes here for more info: https://github.com/paul-gauthier/aider#gpt-4-vs-gpt-35 https://github.com/paul-gauthier/aider#gpt-4-vs-gpt-35 I use GPT for all kinds of busy work, and code quality type work. Adding test cases or quality of life features. Things that I might not have the energy to do myself. GPT can often accomplish these tasks off a 1-2 sentence request. Or if not, it will do all the boilerplate and get you 80% of the way there and it's easy to polish up the final 20%.
NicoJuicy 3y ago
Perhaps it helps: Don't entirely copy-paste your code base. Eg. I adjusted the prompt to the logical separation in my project ( Eg. the module and the task of the class can be derived from it's namespace) and then indexed the codebase through the namespace + classes + method names with their parameters. It helps adding more codebase to the prompt.

rane 3y ago

How have you created the svg screencast in the README?

andrewescott 3y ago
Looking at the source, it appears to have been generated by https://github.com/nbedos/termtosvg https://github.com/nbedos/termtosvg
- anotherpaulg 3y ago
  Yup, that's what I used. It produces really nice, crisp SVG screencasts.

braindead_in 3y ago

I've played around with aider trying to run tests and fix the code, but it just crashes after exceeding the context window. I am now trying to repurpose the AutoGPT example in langchain.

anotherpaulg 3y ago
Sounds like maybe your source files are bigger than the context window? Try including fewer files in the chat. See some notes on GPT models and file sizes here: https://github.com/paul-gauthier/aider#gpt-4-vs-gpt-35 https://github.com/paul-gauthier/aider#gpt-4-vs-gpt-35 I also have some incoming improvements so the tool is more graceful and helpful if you hit the context window limit. And in general, most of my main efforts are focused on making it possible to work with larger and larger codebases in spite of the context window limitations. Please do file an issue with more details on your problems. I can try and help and give you updates if I make improvements to the tool which could solve your use case.

lgas 3y ago

It's interesting to me that you say > Generating new code from whole-cloth seems like an easier task for GPT. as I've been working on a similar tool of my own and I've found the opposite. The initial task I've been trying to have it complete is to build something like https://craigmbooth.com/projects/killer-sudoku-calculator/ https://craigmbooth.com/projects/killer-sudoku-calculator/ but with my bespoke requirements. If I try to specify it up front and have it generate the whole thing it almost always fails in a number of ways at once, despite the requirements being relatively straightforward and clear. However I have had success by walking it through the process step by step along the lines of "please create an index.html that references react from a cdn. It should draw a blank sudoku board" -> "Please add a button that says 'add a cage'. When the button is clicked a box labeled "Cage 1", "Cage 2", etc should be added to a list to the right of the board" -> "Please track the currently selected cage and set the background of the corresponding cage box to green", etc. Likewise, I added the initial set of functions it could access ("Create a File", "Update a File", "Remove a File") manually, but then I had it add additional commands ("Add a directory", "Remove a directory", "Copy a file", etc) and it was able to do it correctly on the first try each time because the pattern already existed.

anotherpaulg 3y ago
I think we agree, but maybe I wasn't writing clearly. You're describing building a green field app starting from nothing, step by step. GPT shines at things like this, because they are by definition small code bases. You can probably fit the whole codebase into the context window. Also, your approach of walking it through step-by-step is perfect, since you get to guide it to build a wise code architecture as you go. It's hard to naively point GPT at a big, existing repo and try and do non-trivial changes to that codebase. Without a bunch of tooling to help it understand the overall codebase, it won't understand or respect the existing modules, abstractions, etc. It will just start trying to write code in a vacuum, which probably isn't the right thing to do when modifying an existing codebase.