4 ms·

Rather than prompting GPT into implementing a solution, can we prompt it to try to preemptively find issues with the codebase or missing-functionality? Also, d

Rather than prompting GPT into implementing a solution, can we prompt it to try to preemptively find issues with the codebase or missing-functionality? Also, do we know what languages GPT-4 "understands" at a sufficient level? What knowledge does it have of post-2021 language features, like in C23?

ftufek 4y ago
It has no post-2021 knowledge, but while playing with it, I found that you can just paste the documentation (no need to even format it) and it'll just "learn" it. For example, safetensors wasn't available back then apparently, I just copied the docs into it and was able to get it write pretty good pytorch code that incorporates safetensors.
- kanyethegreat 4y ago
  i just asked it about safetensors today! also, got a response that amounted to "i don't know what that is. i'm guessing it's X"
  - sharemywin 4y ago
    did you post the documentation into the prompt?
textninja 4y ago
I imagine few shot learning would kick in for most new language features. A feature may be new to a particular language, but is it really new?
- abecedarius 4y ago
  I tried a new Lisp dialect on it, my own hobby language. It could cope well given explanations initially, but with some degradation after a while. The full transcript went to 84kB, so it must be doing some kind of intelligent summarization behind the scenes to stay as coherent as it did, right? (The standard context window is supposed to be 8k tokens.) (https://gist.github.com/darius/b463c7089358fe138a6c29286fe2d37b https://gist.github.com/darius/b463c7089358fe138a6c29286fe2d... paste in painful-to-read format if anyone's really curious. In three parts: intro to language; I ask it to code symbolic differentiation; then a metacircular interpreter.)
  - sharemywin 4y ago
    you might be able to do a 2 step process where you have it only highlight relevant sections of text given your task as an intermediate step. like a filter step before the task step.

ftufek 4y ago

It has no post-2021 knowledge, but while playing with it, I found that you can just paste the documentation (no need to even format it) and it'll just "learn" it. For example, safetensors wasn't available back then apparently, I just copied the docs into it and was able to get it write pretty good pytorch code that incorporates safetensors.

kanyethegreat 4y ago
i just asked it about safetensors today! also, got a response that amounted to "i don't know what that is. i'm guessing it's X"
- sharemywin 4y ago
  did you post the documentation into the prompt?

textninja 4y ago

I imagine few shot learning would kick in for most new language features. A feature may be new to a particular language, but is it really new?

abecedarius 4y ago
I tried a new Lisp dialect on it, my own hobby language. It could cope well given explanations initially, but with some degradation after a while. The full transcript went to 84kB, so it must be doing some kind of intelligent summarization behind the scenes to stay as coherent as it did, right? (The standard context window is supposed to be 8k tokens.) (https://gist.github.com/darius/b463c7089358fe138a6c29286fe2d37b https://gist.github.com/darius/b463c7089358fe138a6c29286fe2d... paste in painful-to-read format if anyone's really curious. In three parts: intro to language; I ask it to code symbolic differentiation; then a metacircular interpreter.)
- sharemywin 4y ago
  you might be able to do a 2 step process where you have it only highlight relevant sections of text given your task as an intermediate step. like a filter step before the task step.