4 ms·

I'm really confused about this. I've been using GPT-4 for coding for months now and it's immensely useful. Sure it makes mistakes; I also make mistakes. Its mis

I'm really confused about this. I've been using GPT-4 for coding for months now and it's immensely useful. Sure it makes mistakes; I also make mistakes. Its mistakes are different from my mistakes. It just feels like it's very very close to being able to close the loop and self-correct incrementally, and once that happens we're dancing on the edge of takeoff. It seems like we're in a situation of "it has the skills but it cannot learn how to reliably invoke them." I just don't think that's a safe place to stand.

trimethylpurine 3y ago
I'll try to clarify and maybe you'll see what I'm getting at. If the code you're writing was written and posted a million times online then ChatGPT is great at regurgitating that and even applying it to more specific applications. No argument. That's pretty cool. It can seem very surreal and give a lot of magic to the card trick. But try this; take a sample of code from the Gio package for Go. I like this example because there's not a lot of published examples for this. The machine would actually have to "think" to accomplish basic work and obviously that's not what it does. Take some example code from their tutorial and just ask ChatGPT 4 to do something simple like change the background black. In my research, 35/37 attempts don't even compile. It tries the same mistakes over and over again. It fails to make reasonable assessment of the compilation errors. The two attempts that did compile; one was a blank white canvas with nothing and the other didn't change anything, it looked exactly like the tutorial example. Something else you can try; tell it to generate a logo and give it some big words for the company name, like "corporation." Watch how it can't even spell. Eventually it will admit that it's not able to do it, which I can only guess was a manual patch to save on operating costs so that you don't keep trying. In short, this is not "intelligence" technology. That's a marketing term. It doesn't do anything remotely close to that and there is no clear path from this technology to that technology. It's just not in the same realm. Maybe the doomsayers have seen some tech that I'm not familiar with, but I am not persuaded by ChatGPT in its current state. I think it will be another tech revolution or maybe two or three before AI leaves the realm of Sci Fi and enters the realm of even theoretical possibility. This tech is machine learning, which is a creative way of saying "near real time statistics." That is really cool in and of itself. But it has nothing to do with "intelligence."
- FeepingCreature 3y ago
  Hm. I've not had any problems using GPT4 with code that it has never seen. But sure, I'll try that... Edit: I took the GIO Hello World sample. I fed it into the GPT-4 API. And I swear to God, first try, first click: https://gist.github.com/FeepingCreature/bde583aa8a2e8e5e1c569190af5aa3f8 https://gist.github.com/FeepingCreature/bde583aa8a2e8e5e1c56... Near-perfect success. One extra import, sure, I'll give it that one, LLMs can't correct when they notice they don't actually need an import in the code. But then it works. Gist link cause I use OpenRouter against the GPT-4 API, maybe that's why? Maybe the "stochastic parrot" vs "oncoming apocalypse" perspective genuinely represents that ..... GPT just doesn't work for some people?? Did they fuck the ChatGPT4 web interface enough somehow so that it's just inept, but somehow not affect the API? Edit: > In my research, 35/37 attempts don't even compile. It tries the same mistakes over and over again. It fails to make reasonable assessment of the compilation errors. Important note: there's a theory that if you've got it making mistakes a few times, it thinks that it's playing "the sort of AI that fucks up a lot" and starts making more mistakes, reinforcing its role. Start over fresh if it seems strangely incapable. Like, it is absolutely possible to use GPT-4 in a way that results in it systematically being incapable of the most basic tasks. These things are not reliably competent; but they're occasionally competent! In my experience, even with completely novel tasks in completely novel environments, the competence is "in there" and can often be elicited with dedicated poking. That's why I think LLMs are enough for a takeoff with enough scale (and basic online learning probably): in my opinion, it's not a matter of attaining the skills but removing the roadblocks. It's like the joke about seeing an old man playing chess with a labrador, and the labrador just carefully picking up figures and moving them around the board, every time in a legal move, and you say "that's amazing, a chess-playing dog!" And the old man scoffs and says, "Nonsense. His endgame is rubbish." Three years ago we didn't think a dog could understand pawn promotion at all. edit: Hey, share your repros? I'm genuinely curious what's going on here now. edit: > Watch how it can't even spell. This is a specific issue with the current generation of LLMs and should be thought of more as a disorder than a fundamental inability. LLMs don't actually ever see letters, they see BPEs, which consist of one up to n letters. For instance, the word "corporation" looks to the AI more like "<27768>". So like- yes. It can not spell. It fundamentally, architecturally, cannot perceive letters in the words you give it; its ability to manually split words into letters is based on chance memorization. Instead, try getting it to split the word into letters in code.

trimethylpurine 3y ago

I'll try to clarify and maybe you'll see what I'm getting at. If the code you're writing was written and posted a million times online then ChatGPT is great at regurgitating that and even applying it to more specific applications. No argument. That's pretty cool. It can seem very surreal and give a lot of magic to the card trick. But try this; take a sample of code from the Gio package for Go. I like this example because there's not a lot of published examples for this. The machine would actually have to "think" to accomplish basic work and obviously that's not what it does. Take some example code from their tutorial and just ask ChatGPT 4 to do something simple like change the background black. In my research, 35/37 attempts don't even compile. It tries the same mistakes over and over again. It fails to make reasonable assessment of the compilation errors. The two attempts that did compile; one was a blank white canvas with nothing and the other didn't change anything, it looked exactly like the tutorial example. Something else you can try; tell it to generate a logo and give it some big words for the company name, like "corporation." Watch how it can't even spell. Eventually it will admit that it's not able to do it, which I can only guess was a manual patch to save on operating costs so that you don't keep trying. In short, this is not "intelligence" technology. That's a marketing term. It doesn't do anything remotely close to that and there is no clear path from this technology to that technology. It's just not in the same realm. Maybe the doomsayers have seen some tech that I'm not familiar with, but I am not persuaded by ChatGPT in its current state. I think it will be another tech revolution or maybe two or three before AI leaves the realm of Sci Fi and enters the realm of even theoretical possibility. This tech is machine learning, which is a creative way of saying "near real time statistics." That is really cool in and of itself. But it has nothing to do with "intelligence."

FeepingCreature 3y ago
Hm. I've not had any problems using GPT4 with code that it has never seen. But sure, I'll try that... Edit: I took the GIO Hello World sample. I fed it into the GPT-4 API. And I swear to God, first try, first click: https://gist.github.com/FeepingCreature/bde583aa8a2e8e5e1c569190af5aa3f8 https://gist.github.com/FeepingCreature/bde583aa8a2e8e5e1c56... Near-perfect success. One extra import, sure, I'll give it that one, LLMs can't correct when they notice they don't actually need an import in the code. But then it works. Gist link cause I use OpenRouter against the GPT-4 API, maybe that's why? Maybe the "stochastic parrot" vs "oncoming apocalypse" perspective genuinely represents that ..... GPT just doesn't work for some people?? Did they fuck the ChatGPT4 web interface enough somehow so that it's just inept, but somehow not affect the API? Edit: > In my research, 35/37 attempts don't even compile. It tries the same mistakes over and over again. It fails to make reasonable assessment of the compilation errors. Important note: there's a theory that if you've got it making mistakes a few times, it thinks that it's playing "the sort of AI that fucks up a lot" and starts making more mistakes, reinforcing its role. Start over fresh if it seems strangely incapable. Like, it is absolutely possible to use GPT-4 in a way that results in it systematically being incapable of the most basic tasks. These things are not reliably competent; but they're occasionally competent! In my experience, even with completely novel tasks in completely novel environments, the competence is "in there" and can often be elicited with dedicated poking. That's why I think LLMs are enough for a takeoff with enough scale (and basic online learning probably): in my opinion, it's not a matter of attaining the skills but removing the roadblocks. It's like the joke about seeing an old man playing chess with a labrador, and the labrador just carefully picking up figures and moving them around the board, every time in a legal move, and you say "that's amazing, a chess-playing dog!" And the old man scoffs and says, "Nonsense. His endgame is rubbish." Three years ago we didn't think a dog could understand pawn promotion at all. edit: Hey, share your repros? I'm genuinely curious what's going on here now. edit: > Watch how it can't even spell. This is a specific issue with the current generation of LLMs and should be thought of more as a disorder than a fundamental inability. LLMs don't actually ever see letters, they see BPEs, which consist of one up to n letters. For instance, the word "corporation" looks to the AI more like "<27768>". So like- yes. It can not spell. It fundamentally, architecturally, cannot perceive letters in the words you give it; its ability to manually split words into letters is based on chance memorization. Instead, try getting it to split the word into letters in code.