3 ms·

If you're expecting to just copy/paste from ChatGPT without understanding what it generates, you're really doing it wrong. I've only used it for basic snippets

If you're expecting to just copy/paste from ChatGPT without understanding what it generates, you're really doing it wrong. I've only used it for basic snippets where I know what the code I need is, but it's faster to describe it than write it out myself (for example, "Using Python and boto3, take all the files in the current directory that end in .csv or .pdf and upload them to a bucket named "reports". Prepend the S3 keys with "YYYY-MM-DD/" using today's date"). For simple things like that, it's been great.

NewsaHackO 2y ago
Yep. OP had to ask it fix mismatched brackets, and is a perfect example of the type of person who definitely should not use LLM for code.
- peteradio 2y ago
  I've seen self described "product owners" claim to build using LLM outputs without knowing how to code.
- boredtofears 2y ago
  I’m sure most people can learn to use LLMs correctly if they can code, it doesn’t matter what “type” of person you are.
- chrisjj 2y ago
  Actually I didn't have to. I chose to. So I could give it another chance to succeed. But do tell. What type of person does not have to ask it to fix its mismatched brackets?
paulmd 2y ago
It’s also constrained by how many examples it’s seen of a particular language or data format, and the general complexity of the format (even without AI you’ll see diff breaking on json files for much the same reason, unless you go out of your way to avoid it - brackets are semantic white noise and there’s nothing unique for it to lock onto between those bracket tokens). At a minimum it’s going to absolutely chew through tokens to maintain that much context. Obscure languages and operating systems are also a weakness - for example I could not get llama3 to reply with FreeBSD commands even after repeatedly telling it not to - there just aren’t that many SO/Reddit/forum posts about BSD in comparison to Linux. And evidently that’s what they’ve trained it on. There isn’t even anything semantically odd about an rc.conf, it spits out correct statements, just for the wrong file in the wrong OS. Everyone acknowledges things like multiplication and counting are hard for LLMs because they aren’t statistically encoded. But counting lisp brackets is also basically the same problem. And unlike YAML it’s not really self-healing or obviously correctable once it’s fucked up (as you noticed). I think the implication is there’s clearly some syntaxes that are much less generateable simply because they require intense amounts of context. (also, as far as counting… I’m sure far smarter people have look at this, but maybe multiplication is something that could benefit from log-addition techniques, that might reduce the complexity quite a bit vs a naive “multiply column by column like a human fourth-grader” approach.)
- chrisjj 2y ago
  > It’s also constrained by how many examples it’s seen of a particular language ChatGPT: "I've encountered numerous examples of Lisp code during my training, covering different applications and techniques within the language. Whether it's simple examples or more complex implementations, I've seen quite a bit of Lisp code." > the general complexity of the format Good then that this example has a very simple format. Here Not a rpoblem in this (even without AI you’ll see diff breaking on json files for much the same reason, unless you go out of your way to avoid it - brackets are semantic white noise
chrisjj 2y ago
? If you're expecting to just copy/paste from ChatGPT without understanding what it generates, you're really doing it wrong. Glad I'm not doing that, then. I understand what it is generating as soon as I try to run it - and get a "missing bracket" error. I've understood that what it has generated is crap. And that only good luck made that easily detectable.

NewsaHackO 2y ago

Yep. OP had to ask it fix mismatched brackets, and is a perfect example of the type of person who definitely should not use LLM for code.

peteradio 2y ago
I've seen self described "product owners" claim to build using LLM outputs without knowing how to code.
boredtofears 2y ago
I’m sure most people can learn to use LLMs correctly if they can code, it doesn’t matter what “type” of person you are.
chrisjj 2y ago
Actually I didn't have to. I chose to. So I could give it another chance to succeed. But do tell. What type of person does not have to ask it to fix its mismatched brackets?

paulmd 2y ago

It’s also constrained by how many examples it’s seen of a particular language or data format, and the general complexity of the format (even without AI you’ll see diff breaking on json files for much the same reason, unless you go out of your way to avoid it - brackets are semantic white noise and there’s nothing unique for it to lock onto between those bracket tokens). At a minimum it’s going to absolutely chew through tokens to maintain that much context. Obscure languages and operating systems are also a weakness - for example I could not get llama3 to reply with FreeBSD commands even after repeatedly telling it not to - there just aren’t that many SO/Reddit/forum posts about BSD in comparison to Linux. And evidently that’s what they’ve trained it on. There isn’t even anything semantically odd about an rc.conf, it spits out correct statements, just for the wrong file in the wrong OS. Everyone acknowledges things like multiplication and counting are hard for LLMs because they aren’t statistically encoded. But counting lisp brackets is also basically the same problem. And unlike YAML it’s not really self-healing or obviously correctable once it’s fucked up (as you noticed). I think the implication is there’s clearly some syntaxes that are much less generateable simply because they require intense amounts of context. (also, as far as counting… I’m sure far smarter people have look at this, but maybe multiplication is something that could benefit from log-addition techniques, that might reduce the complexity quite a bit vs a naive “multiply column by column like a human fourth-grader” approach.)

chrisjj 2y ago
> It’s also constrained by how many examples it’s seen of a particular language ChatGPT: "I've encountered numerous examples of Lisp code during my training, covering different applications and techniques within the language. Whether it's simple examples or more complex implementations, I've seen quite a bit of Lisp code." > the general complexity of the format Good then that this example has a very simple format. Here Not a rpoblem in this (even without AI you’ll see diff breaking on json files for much the same reason, unless you go out of your way to avoid it - brackets are semantic white noise

chrisjj 2y ago

? If you're expecting to just copy/paste from ChatGPT without understanding what it generates, you're really doing it wrong. Glad I'm not doing that, then. I understand what it is generating as soon as I try to run it - and get a "missing bracket" error. I've understood that what it has generated is crap. And that only good luck made that easily detectable.