3 ms·

People use GPT to write computer code. Now that it has "function calling", does that mean it can run a compiler on code and try to fix it based on error message

People use GPT to write computer code. Now that it has "function calling", does that mean it can run a compiler on code and try to fix it based on error messages before showing the code to the user?

mlyle 3y ago
Yes.
- srgpqt 3y ago
  I suppose Rust can finally become a mainstream language, just let the AI deal with massaging our code to appease the compiler.
  - drdaeman 3y ago
    Current LLMs are notoriously bad at design. They can produce code snippets that may even work, but they can't stay focused to make it all fit nicely into a consistent architecture. If you ask it to write you some larger and reasonably complex codebase, you'll end up with a weird junior-grade patchwork. And even if you talk architecture first, they behave as if they have super-short attention spans, so they mostly disregard it and get carried away. (Of course that's not what really happens, I'm kind of anthropomorphising it, but that how it feels like).
    - vidarh 3y ago
      You need to iterate. I've had decent luck having it "fill in the blanks" of a spec, then produce outlines of code with placeholders, then fill in placeholders. Granted, each time with me in the loop to go "hold on, do this slightly differently". So basically treating it as a junior doing the typing and gruntwork, while providing regular feedback on the direction. But you need to effectively constantly refine and narrow down a harness it can be told to work within. But that's often how I work on my own as well, so that's fine.
      - moonchrome 3y ago
        My problem with this is that the interface sucks and the iteration is slow. Supposedly copilot x will have a deeper IDE integration and voice commands.
        vidarh 3y ago
        The iteration is faster than asking a person to do the same tasks. I've started toying with the API to address the interface bit, though.
      - drdaeman 3y ago
        That’s exactly what I did when I’ve experimented with it teaching me how to make a watchOS app (with me having zero iOS and Swift experience), but the issue us that it started to “forget” what we have talked about. It helped me to bootstrap the project and pointed out at some APIs I needed, but then it was a constant fight for code quality. I wanted proper actor abstraction (because I was dealing with Bluetooth, even if I don’t know the language I can see the patterns) and it just loved to write code in style of some cheapest-grade “how to use CoreBluetooth in Swift for dummies” blog articles.
        mlyle 3y ago
        Were you using vanilla ChatGPT or GPT4? Vanilla ChatGPT uses a GPT-3.5turbo version with a context window of 4096 tokens. GPT-4 has twice the context window by default, and has a variant with a context window 8x as big. It's not "8x better" with regards to forgetting what it's doing, but it's substantially better.
        drdaeman 3y ago
        Yes, that was GPT-3.5. GPT-4's 25 messages/3 hours ratelimit wouldn't have led us anywhere, even with me doing my own research and compiling larger lists of change requests would've exhausted that very quickly.
        3y ago
dluc 3y ago
The model doesn't run code, it generates text that happens to be code. It's up to the client calling the model API to use this text, e.g. compiling and executing (if that's your scenario) and calling the model again to fix the original code if needed.
ilaksh 3y ago
You could always take the output and write it to a file, run a compiler on it, and feed the results back into the AI, automatically. This just makes it easier to build systems like that or the system to be more efficient. For example, you can give it a function to call to replace some text in a file. Whereas before you needed to come up with some custom scheme with a diff format or sed call or something which was prone to breaking due to things like minor AI "typos" or newlines etc. This also makes it easy to implement an interface to things like a compiler or other command line tools so the AI can call them itself if necessary. Such as for installing packages or initializing a project. Before you had to keep begging GPT to follow some protocol with file outputs or something. Although GPT-4 was much better (but cost 10X more).

People use GPT to write computer code. Now that it has "function calling", does that mean it can run a compiler on code and try to fix it based on error messages before showing the code to the user?

mlyle 3y ago

Yes.

srgpqt 3y ago
I suppose Rust can finally become a mainstream language, just let the AI deal with massaging our code to appease the compiler.
- drdaeman 3y ago
  Current LLMs are notoriously bad at design. They can produce code snippets that may even work, but they can't stay focused to make it all fit nicely into a consistent architecture. If you ask it to write you some larger and reasonably complex codebase, you'll end up with a weird junior-grade patchwork. And even if you talk architecture first, they behave as if they have super-short attention spans, so they mostly disregard it and get carried away. (Of course that's not what really happens, I'm kind of anthropomorphising it, but that how it feels like).
  - vidarh 3y ago
    You need to iterate. I've had decent luck having it "fill in the blanks" of a spec, then produce outlines of code with placeholders, then fill in placeholders. Granted, each time with me in the loop to go "hold on, do this slightly differently". So basically treating it as a junior doing the typing and gruntwork, while providing regular feedback on the direction. But you need to effectively constantly refine and narrow down a harness it can be told to work within. But that's often how I work on my own as well, so that's fine.
    - moonchrome 3y ago
      My problem with this is that the interface sucks and the iteration is slow. Supposedly copilot x will have a deeper IDE integration and voice commands.
      - vidarh 3y ago
        The iteration is faster than asking a person to do the same tasks. I've started toying with the API to address the interface bit, though.
    - drdaeman 3y ago
      That’s exactly what I did when I’ve experimented with it teaching me how to make a watchOS app (with me having zero iOS and Swift experience), but the issue us that it started to “forget” what we have talked about. It helped me to bootstrap the project and pointed out at some APIs I needed, but then it was a constant fight for code quality. I wanted proper actor abstraction (because I was dealing with Bluetooth, even if I don’t know the language I can see the patterns) and it just loved to write code in style of some cheapest-grade “how to use CoreBluetooth in Swift for dummies” blog articles.
      - mlyle 3y ago
        Were you using vanilla ChatGPT or GPT4? Vanilla ChatGPT uses a GPT-3.5turbo version with a context window of 4096 tokens. GPT-4 has twice the context window by default, and has a variant with a context window 8x as big. It's not "8x better" with regards to forgetting what it's doing, but it's substantially better.
        drdaeman 3y ago
        Yes, that was GPT-3.5. GPT-4's 25 messages/3 hours ratelimit wouldn't have led us anywhere, even with me doing my own research and compiling larger lists of change requests would've exhausted that very quickly.
        3y ago

dluc 3y ago

The model doesn't run code, it generates text that happens to be code. It's up to the client calling the model API to use this text, e.g. compiling and executing (if that's your scenario) and calling the model again to fix the original code if needed.

ilaksh 3y ago

You could always take the output and write it to a file, run a compiler on it, and feed the results back into the AI, automatically. This just makes it easier to build systems like that or the system to be more efficient. For example, you can give it a function to call to replace some text in a file. Whereas before you needed to come up with some custom scheme with a diff format or sed call or something which was prone to breaking due to things like minor AI "typos" or newlines etc. This also makes it easy to implement an interface to things like a compiler or other command line tools so the AI can call them itself if necessary. Such as for installing packages or initializing a project. Before you had to keep begging GPT to follow some protocol with file outputs or something. Although GPT-4 was much better (but cost 10X more).