5 ms·

Problem is generative AI sucks at this. I've had multiple instances where it would say something like "you did this part of the code wrong fix it like this" and

Problem is generative AI sucks at this. I've had multiple instances where it would say something like "you did this part of the code wrong fix it like this" and it would go "sorry for the confusion here's the code with your suggestion " then repeat the initial code or do something else entirely. GPT4, Copilot X, JetBrains AI - I've seen an instance of this problem in all of them. Personally copilot is a magical typing speedup and that got me enthusiastic about next step - but it looks like next step is going to require another breakthrough considering GPT4 HW requirements and actual performance. Chat interface as a search bot is the only use case I've found it useful outside of copilot - regurgitating relevant part of internal documentation as search on steroids, even 3.5 is relatively decent at this.

ZephyrBlu 3y ago
I completely disagree. I've found that GPT-4 and Copilot Chat generate good suggestions most of the time. What kind of things are you try to use codegen for? You also need to provide adequate context. I don't think source code alone is good enough context most of the time (E.g. need an error message as well), unless you're pulling in multiple chunks of code.
- deleted 3y ago
  [deleted]
- gertlex 3y ago
  To me it sounds like what's described is similar to my own interactions with ChatGPT where: 1. Write a prompt, get a full script in response (good! as I expected) 2. Realize something missing in the prompt, or want to improve a specific function. 3. Prompt like "For function foo_bar(), give me just a updated version of that function that adds exception handling for missing files". 4. Chat GPT ignores the (admittedly implied in my language above) "just that function" part, and rewrites the entire script. While usually it only modifies the part of the script I asked it to, it's annoying because (1) it's slower to give me the whole thing back (2) need to do additional checks in case the function I care about depends on things outside of it that might have been changed. I've had this a few times, so I'm in the habit of not trying to iterate with chatGPT when asking it for scripts, but instead just doing the rest myself. (Disclaimers: I haven't tried too hard to find a solution to this; there might be one; I also don't generally combine ChatGPT with Copilot; one or the other on different projects; and haven't tried other forms of using GPT-4...)
  - ZephyrBlu 3y ago
    Are you using GPT-4? I've found it very responsive to follow-up suggestions. I don't re-specify the whole ask though, just ask for the changes like "can you add error handling for missing files?". This is a small recent example: https://chat.openai.com/share/2aa979ca-c796-4dfa-ae13-b21d39194945 https://chat.openai.com/share/2aa979ca-c796-4dfa-ae13-b21d39... And another example where I ask ChatGPT to remove some error handling: https://chat.openai.com/share/90ce6336-f35c-40fc-8fce-baefc59a518d https://chat.openai.com/share/90ce6336-f35c-40fc-8fce-baefc5... GitHub Copilot seems a bit more temperamental. If it doesn't give me something decent first time I don't bother trying to follow-up.
    - gertlex 3y ago
      It blurs together at this point. I've been using GPT-4 for a few months now, was just using 3.5 for months prior to that. I'm also not doing too much, volume-wise; maybe about a script/week. I'll keep this in mind in future work!
vorticalbox 3y ago
"fix this code" is very broad, it will likely do better with more of a prompt. "you will be given a section of code wrapped in ``` your task is analysis this code for any possible issues. You should list any issues you find and why you believe it's an issue. Then explain how to correct it"
- moonchrome 3y ago
  But the context is there, e.g. I get a bad suggestion by LLM - I suggest a fix - it acknowledged the correction and ignores the instructions when generating code.
  - vorticalbox 3y ago
    If you have the ability you can try changing presence_penalty and frequency_penalty.

ZephyrBlu 3y ago

I completely disagree. I've found that GPT-4 and Copilot Chat generate good suggestions most of the time. What kind of things are you try to use codegen for? You also need to provide adequate context. I don't think source code alone is good enough context most of the time (E.g. need an error message as well), unless you're pulling in multiple chunks of code.

deleted 3y ago
[deleted]
gertlex 3y ago
To me it sounds like what's described is similar to my own interactions with ChatGPT where: 1. Write a prompt, get a full script in response (good! as I expected) 2. Realize something missing in the prompt, or want to improve a specific function. 3. Prompt like "For function foo_bar(), give me just a updated version of that function that adds exception handling for missing files". 4. Chat GPT ignores the (admittedly implied in my language above) "just that function" part, and rewrites the entire script. While usually it only modifies the part of the script I asked it to, it's annoying because (1) it's slower to give me the whole thing back (2) need to do additional checks in case the function I care about depends on things outside of it that might have been changed. I've had this a few times, so I'm in the habit of not trying to iterate with chatGPT when asking it for scripts, but instead just doing the rest myself. (Disclaimers: I haven't tried too hard to find a solution to this; there might be one; I also don't generally combine ChatGPT with Copilot; one or the other on different projects; and haven't tried other forms of using GPT-4...)
- ZephyrBlu 3y ago
  Are you using GPT-4? I've found it very responsive to follow-up suggestions. I don't re-specify the whole ask though, just ask for the changes like "can you add error handling for missing files?". This is a small recent example: https://chat.openai.com/share/2aa979ca-c796-4dfa-ae13-b21d39194945 https://chat.openai.com/share/2aa979ca-c796-4dfa-ae13-b21d39... And another example where I ask ChatGPT to remove some error handling: https://chat.openai.com/share/90ce6336-f35c-40fc-8fce-baefc59a518d https://chat.openai.com/share/90ce6336-f35c-40fc-8fce-baefc5... GitHub Copilot seems a bit more temperamental. If it doesn't give me something decent first time I don't bother trying to follow-up.
  - gertlex 3y ago
    It blurs together at this point. I've been using GPT-4 for a few months now, was just using 3.5 for months prior to that. I'm also not doing too much, volume-wise; maybe about a script/week. I'll keep this in mind in future work!

vorticalbox 3y ago

"fix this code" is very broad, it will likely do better with more of a prompt. "you will be given a section of code wrapped in ``` your task is analysis this code for any possible issues. You should list any issues you find and why you believe it's an issue. Then explain how to correct it"

moonchrome 3y ago
But the context is there, e.g. I get a bad suggestion by LLM - I suggest a fix - it acknowledged the correction and ignores the instructions when generating code.
- vorticalbox 3y ago
  If you have the ability you can try changing presence_penalty and frequency_penalty.