4 ms·

The problem with ChatGPT is that it often reads authoritative. But is often just flat out wrong. I asked it a few questions for which I consider myself a subje

The problem with ChatGPT is that it often reads authoritative. But is often just flat out wrong. I asked it a few questions for which I consider myself a subject matter expert and the answers were laughably wrong.

photochemsyn 4y ago
I just asked it to provide a specific example of a security breach caused by a Java serialization flaw and it just made something up, i.e. it found some random security breach (unrelated to anything Java) and found some other random Java serialization bug and claimed that breach was due to that bug. A few minutes on Google revealed they were completely unrelated. Ouch.
- drusepth 4y ago
  It seems like if the contextual analogy here was carried through and ChatGPT were to leave its message as a comment on a Java-security thread, the same approach would apply (a few minutes of research on Google) and the provably-incorrect message would either be downvoted or commented on, just like a human comment with the same content would be.
63 4y ago
Agreed. I chatted with it about the Chinese remainder theorem today and it gave me an example that didn't work and then insisted that 30 is not divisible by 2 when I questioned it. This was simple enough to spot but I was halfway through the example by the time I realized I couldn't trust it at all. Its confidence is annoying.
- kderbyma 4y ago
  It told me the longest known prime number ended with a 2. Definitely not accurate when it comes to math.
  - tobtahcia 4y ago
    ChatGPT is not a calculator, it's a language model.
toofy 4y ago
i’ve seen so many examples of this over the past few weeks. just thinking about how many people will just eat what it feeds them borders on terrifying. in so many instances, it’s just wrong but continues on so confidently. one of the things i find most interesting is that it has no idea when it’s wrong so it just keeps going. we already have a fairly significant growing problem of people who refuse to admit (even to themselves) what they don’t know, and now this will just exacerbate the problem. it’s like the worst of debateBro culture has just been automated.
- SoftTalker 4y ago
  > in so many instances, it’s just wrong but continues on so confidently Sounds sociopathic, and also like many politicians and people in leadership positions.
culanuchachamim 4y ago
For me it happend when I asked to write a function using BigQuery, it wrote a function that made a lot of sense but was wrong, because the command didn't exist in BigQuery. When I replay that the function didn't work, it told me, something like this: You right the function that I used it was only working on beta mode, now you have to use the following.... And again it was wrong. I made a little research and never was such a beta commandm.. And then I got that it just makes up things that it don't know, but says it with authority.
- scarface74 4y ago
  I asked it to write a function in Python that would return the list of AWS accounts in an organization with a given tag key and value. The code looked right, initialized boto3 correctly and called a function on it get_account_numbers_by_tag on the organizations object. I wondered why I never heard of that function and nor did I find it when searching. Turns out, there is no such function.
  - lhuser123 4y ago
    It sounds a lot similar to normal thinking errors that we make.
    - scarface74 4y ago
      The second time, it gave me code that was almost right. Just now I asked Write a Python script that returns all of the accounts in an AWS organization with a given tag where the user specifies the tag key and value using command line arguments I thought the code had to be wrong because it used concepts I had never heard of. This time it used the resource group API. I have never heard of the API. But it does exist. I also couldn’t find sample code on the internet that did anything similar. But from looking at the documentation it should work. I learned something new today. BTW, for context when I claimed to be a “subject matter expert” above, I work at AWS in Professional Services, code most days using the AWS API and I would have never thought of the solution it gave me.
  - 4y ago
oezi 4y ago
I guess after cookie banners this is going to be the next frontier of regulation: prohibiting AIs to lie to humans.
khyryk 4y ago
My biggest fear for the short term is that tools like ChatGPT would allow spamming most of the internet with the equivalent of a Gish Gallop -- so much plausible-looking bullshit spewed out in a short time that it would be a lost cause to attempt to sort through it.
- nonrandomstring 4y ago
  If content known to be of human origin could be archived and frozen now in late 2022 it may become valuable in a few years. Some kind of verifiably timestamped encryption might be useful.
tstrimple 4y ago
> The problem with ChatGPT is that it often reads authoritative. But is often just flat out wrong. That sounds pretty damn human to me.
- tobtahcia 4y ago
  yeah, tough to distinguish human BS, from an incorrect ChatGPT.

photochemsyn 4y ago

I just asked it to provide a specific example of a security breach caused by a Java serialization flaw and it just made something up, i.e. it found some random security breach (unrelated to anything Java) and found some other random Java serialization bug and claimed that breach was due to that bug. A few minutes on Google revealed they were completely unrelated. Ouch.

drusepth 4y ago
It seems like if the contextual analogy here was carried through and ChatGPT were to leave its message as a comment on a Java-security thread, the same approach would apply (a few minutes of research on Google) and the provably-incorrect message would either be downvoted or commented on, just like a human comment with the same content would be.

63 4y ago

Agreed. I chatted with it about the Chinese remainder theorem today and it gave me an example that didn't work and then insisted that 30 is not divisible by 2 when I questioned it. This was simple enough to spot but I was halfway through the example by the time I realized I couldn't trust it at all. Its confidence is annoying.

kderbyma 4y ago
It told me the longest known prime number ended with a 2. Definitely not accurate when it comes to math.
- tobtahcia 4y ago
  ChatGPT is not a calculator, it's a language model.

toofy 4y ago

i’ve seen so many examples of this over the past few weeks. just thinking about how many people will just eat what it feeds them borders on terrifying. in so many instances, it’s just wrong but continues on so confidently. one of the things i find most interesting is that it has no idea when it’s wrong so it just keeps going. we already have a fairly significant growing problem of people who refuse to admit (even to themselves) what they don’t know, and now this will just exacerbate the problem. it’s like the worst of debateBro culture has just been automated.

SoftTalker 4y ago
> in so many instances, it’s just wrong but continues on so confidently Sounds sociopathic, and also like many politicians and people in leadership positions.

culanuchachamim 4y ago

For me it happend when I asked to write a function using BigQuery, it wrote a function that made a lot of sense but was wrong, because the command didn't exist in BigQuery. When I replay that the function didn't work, it told me, something like this: You right the function that I used it was only working on beta mode, now you have to use the following.... And again it was wrong. I made a little research and never was such a beta commandm.. And then I got that it just makes up things that it don't know, but says it with authority.

scarface74 4y ago
I asked it to write a function in Python that would return the list of AWS accounts in an organization with a given tag key and value. The code looked right, initialized boto3 correctly and called a function on it get_account_numbers_by_tag on the organizations object. I wondered why I never heard of that function and nor did I find it when searching. Turns out, there is no such function.
- lhuser123 4y ago
  It sounds a lot similar to normal thinking errors that we make.
  - scarface74 4y ago
    The second time, it gave me code that was almost right. Just now I asked Write a Python script that returns all of the accounts in an AWS organization with a given tag where the user specifies the tag key and value using command line arguments I thought the code had to be wrong because it used concepts I had never heard of. This time it used the resource group API. I have never heard of the API. But it does exist. I also couldn’t find sample code on the internet that did anything similar. But from looking at the documentation it should work. I learned something new today. BTW, for context when I claimed to be a “subject matter expert” above, I work at AWS in Professional Services, code most days using the AWS API and I would have never thought of the solution it gave me.
- 4y ago

oezi 4y ago

I guess after cookie banners this is going to be the next frontier of regulation: prohibiting AIs to lie to humans.

khyryk 4y ago

My biggest fear for the short term is that tools like ChatGPT would allow spamming most of the internet with the equivalent of a Gish Gallop -- so much plausible-looking bullshit spewed out in a short time that it would be a lost cause to attempt to sort through it.

nonrandomstring 4y ago
If content known to be of human origin could be archived and frozen now in late 2022 it may become valuable in a few years. Some kind of verifiably timestamped encryption might be useful.

tstrimple 4y ago

> The problem with ChatGPT is that it often reads authoritative. But is often just flat out wrong. That sounds pretty damn human to me.

tobtahcia 4y ago
yeah, tough to distinguish human BS, from an incorrect ChatGPT.