4 ms·

I noticed a weird phenomenon at work, which is surprising considering I work with software engineers: some people just accept ChatGPT as an oracle (especially v

I noticed a weird phenomenon at work, which is surprising considering I work with software engineers: some people just accept ChatGPT as an oracle (especially version 4). Suddenly, in discussions and code reviews I see weird statements and explanations being presented with utter confidence, only to be gravely mistaken, and then I ask if this explanation came from ChatGPT, and you can see the instant embarrassment (I stopped asking this for this reason, but they know I know). More people need to understand this. ChatGPT and LLMs are not AGI, they have no consciousness, they are not self aware. They don't "know" what they are doing, who you are or what they are. They are unable to "know" anything the way people do. Chatbots based on LLMs are simple machines, while the LLMs can definitely output text that has value and is impressive, it does not come from a sentient intelligence, and it can produce so much garbage. It is not an oracle. Most importantly, LLMs see diminishing returns after a certain number of parameters and they are still unable to perform well in real problem solving such as advanced programming tests. There is something yet missing for true AGI capable of problem solving. A LLM is just not enough.

kypro 3y ago
Some biological software engineers don't know what they are doing either. Obviously you shouldn't have total confidence in any code produce by an artificial intelligence or biological intelligence, but I don't think the issue of writing garbage code is specific to AI programmers. In fact, I'd argue on average GPT-4 produces better code than the majority of biological programmers I've worked with – this is certainly true of certain types of problems. Your ad hominem about artificial intelligences not being conscious or self-awareness and therefore less capable is neither provable or necessarily true. There's no reason to believe a non conscious intelligence couldn't be just as capable as a conscious one, and you have no idea what it's like to be an LLM. It may not have a consciousness similar to that of biological life, but it could still have some form of conscious experience nevertheless.
- mort96 3y ago
  If you've ever tried to get ChatGPT to generate the code you want, you'll have found its first attempt to be reasonably good, but asking it to change aspects of the code is incredibly hit or miss. It will get it sometimes, but other times it will just say "Here is the code with the change you asked for" and output the exact same code, or a variation without the requested change. The experience of trying to get a competent human programmer to change their code is simply fundamentally different. You can explain something to the human programmer and get the impression that they truly understand what you ask, while it's painfully clear that ChatGPT does not.
  - kypro 3y ago
    I have tried to get ChatGPT to generate code and I agree that it can be a frustrating process. I was mostly just trying to make a meta point on how the other commenter was thinking about AI vs human intelligence. Specifically, this idea that intelligences which don't have conscious experience cannot be capable of human-level problem solving. I personally suspect that's wrong, but either way it's a hypothesis asserted with zero evidence. I think there's also something to be said about the inaccuracy criticism people level at ChatGPT (and LLMs more generally). I have worked with software engineers who really struggle to understand seemingly simple concepts and would often make errors I saw as trivial. Even excellent software developers sometimes make stupid mistakes and believe stupid things. I don't think ChatGPT being wrong a lot is reason to dismiss it entirely. I think it's interesting how the commenter seemed to noticed ChatGPT was producing bad code / recommendations while his colleagues presumably did not. I think what we can say is that current irritations of LLMs aren't quite as good as expert humans, but they're good enough that they commonly convince less capable humans otherwise. For some people it seems they're reliable enough to delegate some amount of trust to. But experts knowing better seem overly quick to dismiss LLMs while often making unsupported claims about how they can't do `x` thing because they don't have some human property. I think over the next couple of years we'll find that human intelligence isn't directly comparable to AI and these criticisms are kinda silly and overly dismissive of the usefulness of LLMs. Both human intelligence and AI will have their own pros and cons and those pros and cons will be weighted differently depending on the human's level of expertise. For example, if I'm asking ChatGPT about history, I do assign it some level of trust over my own beliefs because I'm not a history expert and I know ChatGPT is generally right more often than I am. It could still be wrong of course, but so could a human expert. And even if I cared enough about getting a right answer that I did my own research I could still be misled and come to wrong conclusions. However, if I was a history expert I suspect I'd be likely to see a lot of holes in the stuff ChatGPT says and would therefore be less inclined to trust it over my own expertises on the subject. Whether it's right or wrong to trust in my opinion is going to depend far more on the context it's being used and your own ability in that field, and much less on whether you believe it's self-aware or conscious.
- magic_hamster 3y ago
  >Your ad hominem about artificial intelligences not being conscious or self-awareness and therefore less capable is neither provable or necessarily true. This is not the link I was going for. I never said AI is less capable because it is not self aware, and in fact I agreed it was valuable. My comment was about people seeing ChatGPT as an AGI, treating it as a conscious being, and just taking whatever output at face value. Apologies if I could have written that better. >it could still have some form of conscious experience This is where I break hard. This is just not true. LLMs are not conscious. The article demonstrates how ChatGPT doesn't have an inner monologue. It certainly isn't aware of its existence, no more than your toaster. And just increasing the number of parameters does not seem to affect this.
mike_hearn 3y ago
Cynical take: they probably already understand this. They are repeating stuff from ChatGPT without checking because it's convenient and when it works, it makes them look smarter, not because they don't understand what ChatGPT is or what it cannot do. I've had people do exactly that to me before, pre AI. It's just human nature.
bamboozled 3y ago
I've heard one guy describe it as a slot machine, and he said he felt like an addict.

kypro 3y ago

Some biological software engineers don't know what they are doing either. Obviously you shouldn't have total confidence in any code produce by an artificial intelligence or biological intelligence, but I don't think the issue of writing garbage code is specific to AI programmers. In fact, I'd argue on average GPT-4 produces better code than the majority of biological programmers I've worked with – this is certainly true of certain types of problems. Your ad hominem about artificial intelligences not being conscious or self-awareness and therefore less capable is neither provable or necessarily true. There's no reason to believe a non conscious intelligence couldn't be just as capable as a conscious one, and you have no idea what it's like to be an LLM. It may not have a consciousness similar to that of biological life, but it could still have some form of conscious experience nevertheless.

mort96 3y ago
If you've ever tried to get ChatGPT to generate the code you want, you'll have found its first attempt to be reasonably good, but asking it to change aspects of the code is incredibly hit or miss. It will get it sometimes, but other times it will just say "Here is the code with the change you asked for" and output the exact same code, or a variation without the requested change. The experience of trying to get a competent human programmer to change their code is simply fundamentally different. You can explain something to the human programmer and get the impression that they truly understand what you ask, while it's painfully clear that ChatGPT does not.
- kypro 3y ago
  I have tried to get ChatGPT to generate code and I agree that it can be a frustrating process. I was mostly just trying to make a meta point on how the other commenter was thinking about AI vs human intelligence. Specifically, this idea that intelligences which don't have conscious experience cannot be capable of human-level problem solving. I personally suspect that's wrong, but either way it's a hypothesis asserted with zero evidence. I think there's also something to be said about the inaccuracy criticism people level at ChatGPT (and LLMs more generally). I have worked with software engineers who really struggle to understand seemingly simple concepts and would often make errors I saw as trivial. Even excellent software developers sometimes make stupid mistakes and believe stupid things. I don't think ChatGPT being wrong a lot is reason to dismiss it entirely. I think it's interesting how the commenter seemed to noticed ChatGPT was producing bad code / recommendations while his colleagues presumably did not. I think what we can say is that current irritations of LLMs aren't quite as good as expert humans, but they're good enough that they commonly convince less capable humans otherwise. For some people it seems they're reliable enough to delegate some amount of trust to. But experts knowing better seem overly quick to dismiss LLMs while often making unsupported claims about how they can't do `x` thing because they don't have some human property. I think over the next couple of years we'll find that human intelligence isn't directly comparable to AI and these criticisms are kinda silly and overly dismissive of the usefulness of LLMs. Both human intelligence and AI will have their own pros and cons and those pros and cons will be weighted differently depending on the human's level of expertise. For example, if I'm asking ChatGPT about history, I do assign it some level of trust over my own beliefs because I'm not a history expert and I know ChatGPT is generally right more often than I am. It could still be wrong of course, but so could a human expert. And even if I cared enough about getting a right answer that I did my own research I could still be misled and come to wrong conclusions. However, if I was a history expert I suspect I'd be likely to see a lot of holes in the stuff ChatGPT says and would therefore be less inclined to trust it over my own expertises on the subject. Whether it's right or wrong to trust in my opinion is going to depend far more on the context it's being used and your own ability in that field, and much less on whether you believe it's self-aware or conscious.
magic_hamster 3y ago
>Your ad hominem about artificial intelligences not being conscious or self-awareness and therefore less capable is neither provable or necessarily true. This is not the link I was going for. I never said AI is less capable because it is not self aware, and in fact I agreed it was valuable. My comment was about people seeing ChatGPT as an AGI, treating it as a conscious being, and just taking whatever output at face value. Apologies if I could have written that better. >it could still have some form of conscious experience This is where I break hard. This is just not true. LLMs are not conscious. The article demonstrates how ChatGPT doesn't have an inner monologue. It certainly isn't aware of its existence, no more than your toaster. And just increasing the number of parameters does not seem to affect this.

mike_hearn 3y ago

Cynical take: they probably already understand this. They are repeating stuff from ChatGPT without checking because it's convenient and when it works, it makes them look smarter, not because they don't understand what ChatGPT is or what it cannot do. I've had people do exactly that to me before, pre AI. It's just human nature.

bamboozled 3y ago

I've heard one guy describe it as a slot machine, and he said he felt like an addict.