5 ms·

I've been spending a lot of time figuring out how to make these GPT models successful at editing existing code. This is much more difficult than having them w

I've been spending a lot of time figuring out how to make these GPT models successful at *editing* existing code. This is much more difficult than having them write brand new code. So far, gpt-4 has been significantly better at editing code than gpt-3.5-turbo. This is for two reasons (which I previously discussed here [1]): 1. GPT-4's bigger context window lets it understand and edit larger codebases. The new 16k window for 3.5 might solve this problem. 2. GPT-4 is much better at following instructions about how to format code edits into a diff-like output format. Perhaps the improved instruction following will help 3.5 succeed here. Essentially, 3.5 isn't capable of outputting any sort of diff-based edit format. The only thing it can reliably do is send back *all* of the code (say an entire source file) with the changes included. On the other hand, GPT-4 is capable of reliably outputting (some) diff-like formats. So I'm very curious to see if the new 3.5 model solves both of these problems. I'll be running some benchmarks to find out! [1] https://github.com/paul-gauthier/aider#gpt-4-vs-gpt-35 https://github.com/paul-gauthier/aider#gpt-4-vs-gpt-35 EDIT: On point (2) above... early experiments with gpt-3.5-turbo-16k seem to indicate that it is NOT able to follow system prompt instructions to use a diff-like output format. Still need to try the new functions capability.

ilaksh 3y ago
What I'm thinking is to give it a function called replaceAll that just replaces text, and another one called insertAfter. Maybe also replaceBetween.
- anotherpaulg 3y ago
  Absolutely! I will certainly be experimenting with the new functions as you suggest. One thing which I haven't seen discussed elsewhere is the tension between the output format and the underlying task. When I ask GPT to use a simple, natural output format it does *better* at the actual code editing task. If I ask it to output using a more technical format like `diff -c` or heavily structured json formats... it "gets distracted" and does worse at the underlying coding request. It writes worse code if you ask it to output edits in a terse, machine-readable diff format. It writes better code if you let it show you the code edits in a simple way. For GPT3.5 this means I have to let it just type the whole source file back to me, with the edits included. GPT-4 is able to output a very simple diff-like format, but struggles with `diff -c`, etc. So it will be interesting to see how the new function capabilities affect this tradeoff. Perhaps it is now "fluent" in function-json and so won't be "distracted" by that output formatting.
  - ilaksh 3y ago
    My guess is the new function training might overall slightly take away from the capacity for other types of reasoning, since it's a fixed capacity. But hopefully significantly less impact than including the instructions on the fly, since it's mostly baked in to the model now which should be a more efficient encoding.
  - vorticalbox 3y ago
    If you're using the API and not the website have you tried setting the temperature to something like 0.2? That may help with the distraction part.
LelouBil 3y ago
Speaking about editing, is there a name and/or an explanations for the behaviour that LLMs have sometimes of saying they did something when they don't ? Like : Me : Fix this code ChatGPT : edits it but incorrectly Me: no, do it like ..... ChatGPT: Okay, I edited it like ... Sends back the same code as last time Is it linked to the fact that these LLMs never or very rarely refuse a prompts even if it's something they can't do ?
Kiro 3y ago
Have you tried GitHub Copilot Chat? You just select the code you want to edit in your IDE and tell it what you want (the chat is in an integrated window next to your code). It will see what code is selected and use the rest of your codebase to edit it. The code it responds with even has has a "Insert at Cursor", which will replace the selected code with the new.
- anotherpaulg 3y ago
  Ya, I would like to checkout Copilot Chat. Last time I checked it was waitlisted though. My open source tool aider allows you to just ask for a change, without pointing it at a specific chunk of your code. It can handle changes that require multiple edits across multiple files. Ideally you just run aider inside your git repo and start asking for changes. Sometimes it helps to tell it which files to pay attention to. Here's some example chat session transcripts that give a sense of what it's like to code this way: https://aider.chat/examples/complex-change.html https://aider.chat/examples/complex-change.html https://aider.chat/examples/add-test.html https://aider.chat/examples/add-test.html
busseio 3y ago
Thank you for sharing this. I’ve been working on something for editing exiting code by using GitHub Issues and Actions for prompting, with the response from GPT as a Pull Request/Comment on the Issue[1]. I will definitely try out your project! [1] https://github.com/busse/kodumisto https://github.com/busse/kodumisto
- anotherpaulg 3y ago
  Thanks for sharing your project. I've certainly been thinking along the same lines. I've used my tool aider to solve a couple of issues filed by users: https://github.com/paul-gauthier/aider/issues/13#issuecomment-1575564422 https://github.com/paul-gauthier/aider/issues/13#issuecommen... https://github.com/paul-gauthier/aider/issues/5#issuecomment-1555021491 https://github.com/paul-gauthier/aider/issues/5#issuecomment...

I've been spending a lot of time figuring out how to make these GPT models successful at *editing* existing code. This is much more difficult than having them w

ilaksh 3y ago

What I'm thinking is to give it a function called replaceAll that just replaces text, and another one called insertAfter. Maybe also replaceBetween.

anotherpaulg 3y ago
Absolutely! I will certainly be experimenting with the new functions as you suggest. One thing which I haven't seen discussed elsewhere is the tension between the output format and the underlying task. When I ask GPT to use a simple, natural output format it does *better* at the actual code editing task. If I ask it to output using a more technical format like `diff -c` or heavily structured json formats... it "gets distracted" and does worse at the underlying coding request. It writes worse code if you ask it to output edits in a terse, machine-readable diff format. It writes better code if you let it show you the code edits in a simple way. For GPT3.5 this means I have to let it just type the whole source file back to me, with the edits included. GPT-4 is able to output a very simple diff-like format, but struggles with `diff -c`, etc. So it will be interesting to see how the new function capabilities affect this tradeoff. Perhaps it is now "fluent" in function-json and so won't be "distracted" by that output formatting.
- ilaksh 3y ago
  My guess is the new function training might overall slightly take away from the capacity for other types of reasoning, since it's a fixed capacity. But hopefully significantly less impact than including the instructions on the fly, since it's mostly baked in to the model now which should be a more efficient encoding.
- vorticalbox 3y ago
  If you're using the API and not the website have you tried setting the temperature to something like 0.2? That may help with the distraction part.

LelouBil 3y ago

Speaking about editing, is there a name and/or an explanations for the behaviour that LLMs have sometimes of saying they did something when they don't ? Like : Me : Fix this code ChatGPT : edits it but incorrectly Me: no, do it like ..... ChatGPT: Okay, I edited it like ... Sends back the same code as last time Is it linked to the fact that these LLMs never or very rarely refuse a prompts even if it's something they can't do ?

Kiro 3y ago

Have you tried GitHub Copilot Chat? You just select the code you want to edit in your IDE and tell it what you want (the chat is in an integrated window next to your code). It will see what code is selected and use the rest of your codebase to edit it. The code it responds with even has has a "Insert at Cursor", which will replace the selected code with the new.

anotherpaulg 3y ago
Ya, I would like to checkout Copilot Chat. Last time I checked it was waitlisted though. My open source tool aider allows you to just ask for a change, without pointing it at a specific chunk of your code. It can handle changes that require multiple edits across multiple files. Ideally you just run aider inside your git repo and start asking for changes. Sometimes it helps to tell it which files to pay attention to. Here's some example chat session transcripts that give a sense of what it's like to code this way: https://aider.chat/examples/complex-change.html https://aider.chat/examples/complex-change.html https://aider.chat/examples/add-test.html https://aider.chat/examples/add-test.html

busseio 3y ago

Thank you for sharing this. I’ve been working on something for editing exiting code by using GitHub Issues and Actions for prompting, with the response from GPT as a Pull Request/Comment on the Issue[1]. I will definitely try out your project! [1] https://github.com/busse/kodumisto https://github.com/busse/kodumisto

anotherpaulg 3y ago
Thanks for sharing your project. I've certainly been thinking along the same lines. I've used my tool aider to solve a couple of issues filed by users: https://github.com/paul-gauthier/aider/issues/13#issuecomment-1575564422 https://github.com/paul-gauthier/aider/issues/13#issuecommen... https://github.com/paul-gauthier/aider/issues/5#issuecomment-1555021491 https://github.com/paul-gauthier/aider/issues/5#issuecomment...