6 ms·

I threw a challenging rendering problem at it and I was pretty impressed with the overall structure and implementation. But as I looked deeper, the flaws became

I threw a challenging rendering problem at it and I was pretty impressed with the overall structure and implementation. But as I looked deeper, the flaws became apparent. It simply made up APIs that didn’t exist, and when prompted to fix it, couldn’t figure it out. Still, despite being fundamentally wrong it did send me down some different paths.

splatzone 4y ago
I’ve had very good results from running the code and pasting the errors back into ChatGPT and asking it what to do. Sometimes it corrects itself quite well
- behnamoh 4y ago
  Put that in a loop and see if AGI emerges.
rel2thr 4y ago
yea I find it to be wrong a lot when coding But its faster for me to fix existing code than to write code from scratch so its still better than nothing for me
- osteele 4y ago
  Same. It seems similar to Copilot in that regard, but better at text-to-code, porting between languages or frameworks, and generating test cases and readmes: https://notes.osteele.com/gpt-experiments/using-chatgpt-to-port-a-python-package-to-javascript https://notes.osteele.com/gpt-experiments/using-chatgpt-to-p...
jasfi 4y ago
Using APIs that don't exist is the biggest problem I've seen with ChatGPT, and it seems GPT-4 as well.
- vitorgrs 4y ago
  One of the advantages of Bing, and do guess now ChatGPT with browsing plugin, is that it's able to search on the web for the right API.
- saulpw 4y ago
  To be fair, using APIs that I think should exist, is how I develop most of my APIs.
  - jasfi 4y ago
    Except that I wasn't asking it to develop a new API.
    - saulpw 4y ago
      Then I guess you're not using it to its fullest potential ;)
      - bamboozled 4y ago
        We can’t keep blaming the prompter.
    - brabel 4y ago
      It's very likely it was using other languages' as "inspiration" given there's very little Zig code out there... so it's maybe natural it would use APIs that don't yet exist... perhaps informing it that it also needs to implement those APIs could work?
- exodontist 4y ago
  I asked chatgpt about the api for an old programming game called chipwits.. it invented a whole programming language that it called chiptalk with an amalgam of the original chipwits stuff, missing some bits and adding others, and generated a parser for it, which I implemented and got to work, before figuring out how much was imaginary, after talking to the original chipwits devs. They found it pretty amusing.
deleted 4y ago
[deleted]
whitehexagon 4y ago
>It simply made up APIs that didn’t exist That has been my experience with Zig. It led me to the conclusion that there are just too many 'non indexed' developer tools in use these days, so there isnt much training data for new topics. But it was happy to hallucinate API's and their proof of existence.

splatzone 4y ago

I’ve had very good results from running the code and pasting the errors back into ChatGPT and asking it what to do. Sometimes it corrects itself quite well

behnamoh 4y ago
Put that in a loop and see if AGI emerges.

rel2thr 4y ago

yea I find it to be wrong a lot when coding But its faster for me to fix existing code than to write code from scratch so its still better than nothing for me

osteele 4y ago
Same. It seems similar to Copilot in that regard, but better at text-to-code, porting between languages or frameworks, and generating test cases and readmes: https://notes.osteele.com/gpt-experiments/using-chatgpt-to-port-a-python-package-to-javascript https://notes.osteele.com/gpt-experiments/using-chatgpt-to-p...

jasfi 4y ago

Using APIs that don't exist is the biggest problem I've seen with ChatGPT, and it seems GPT-4 as well.

vitorgrs 4y ago
One of the advantages of Bing, and do guess now ChatGPT with browsing plugin, is that it's able to search on the web for the right API.
saulpw 4y ago
To be fair, using APIs that I think should exist, is how I develop most of my APIs.
- jasfi 4y ago
  Except that I wasn't asking it to develop a new API.
  - saulpw 4y ago
    Then I guess you're not using it to its fullest potential ;)
    - bamboozled 4y ago
      We can’t keep blaming the prompter.
  - brabel 4y ago
    It's very likely it was using other languages' as "inspiration" given there's very little Zig code out there... so it's maybe natural it would use APIs that don't yet exist... perhaps informing it that it also needs to implement those APIs could work?
exodontist 4y ago
I asked chatgpt about the api for an old programming game called chipwits.. it invented a whole programming language that it called chiptalk with an amalgam of the original chipwits stuff, missing some bits and adding others, and generated a parser for it, which I implemented and got to work, before figuring out how much was imaginary, after talking to the original chipwits devs. They found it pretty amusing.

deleted 4y ago

[deleted]

whitehexagon 4y ago

>It simply made up APIs that didn’t exist That has been my experience with Zig. It led me to the conclusion that there are just too many 'non indexed' developer tools in use these days, so there isnt much training data for new topics. But it was happy to hallucinate API's and their proof of existence.