10 ms·

> Please stop assuming anything it says it true. This is overstated and easily disproved. ChatGPT produces accurate facts as a matter of course. test it right

> Please stop assuming anything it says it true. This is overstated and easily disproved. ChatGPT produces accurate facts as a matter of course. test it right now: "How tall is the Statue of Liberty?", "In what state was Abraham Lincoln born?", etc. There are an infinite list of factual questions for which it produces accurate and correct answers. It loves to hallucinate api methods that don't exist. It can struggle with indivudual logic problems or questions. But these limitations have clear explanations. It's doing language completion so it will infer probable things that don't actually exist. It wasn't designed to do logic problems so it will struggle with classes of them. Dismissing it as completely unreliable is unnecessary hyperbole. It's a software tool that has strengths, weaknesses and limitations. Like with any other software, it's up to us to learn what those are and how it can be used to make useful products.

sarchertech 4y ago
It’s completely unreliable because it sounds confident and plausible even when it’s wrong. Essentially it fails in ways that are very hard to detect. If you ask it question you have no way to know whether it’s right or not unless you already know the answer. I’ve asked it many questions about my area of expertise, distributed systems, and it was often flat out wrong, but to a non expert it sounded perfectly plausible. I asked my my wife, a physician, to try it out and she reported the exact same problem. Some of the answers it gave had even veered off from just annoying to actually dangerous. That doesn’t mean it can’t be useful, but using it to answer important questions or using it to teach yourself something you don’t already know is dangerous.
- beefield 4y ago
  > It’s completely unreliable because it sounds confident and plausible even when it’s wrong This describes pretty accurately a non-trivial amount of people I have worked with.
  - sarchertech 4y ago
    And hopefully after someone shows you that they can be described that way, you stop trusting what they say (at least about the thing they are known to be unreliable about).
    - vidarh 4y ago
      A whole lot of people can be trusted within a reasonable identified set of constraint, and we do, while discounting them in others where they're known to spew bullshit. It's very rarely all or nothing.
      - sarchertech 4y ago
        Obviously. Chatgpt will randomly spew bullshit about nearly any topic though. So you can really only trust it for things you are already an expert in or things that are very easy to verify.
        williamcotton 4y ago
        It's not random. It's probabilistic. There is a very big difference. The types of errors are predictable which means they can be improved, and empirically so if you follow the academic literature.
        sarchertech 4y ago
        I have read enough of the academic literature to say with some confidence that the majority of the errors I’m talking about are not probabilistic in any meaningful sense. At least not in a way that can be predicted from the amount of readily available training data.
        zmgsabst 4y ago
        Ironically, this reply is about as useful as if chatGPT wrote it: - there’s no cited facts - it boldly proclaims its conclusion - which you’d only be able to verify as an expert …so I’m having trouble understanding what you’re complaining about with chatGPT, when that seems to be the standard for discourse.
  - peoplearepeople 4y ago
    You shouldn't listen to them either then
  - discreteevent 4y ago
    It may describe a number of people you have worked with but it does not describe the average medical textbook which will usually describe the best knowledge we have of a condition. If chatgpt produces something that looks like it came from a medical textbook then it is hard to blame many people for believing it. More generally, people started to trust computers more than people as soon as calculators demonstrated their reliability.
    - lordnacho 4y ago
      I don't know, I get this feeling ChatGPT has also read all the quack books where the condition depends on the alignment of stars when you were born, or how chicken entrails land when the shaman "studies" it. Those books are also written confidently without giving any sign of being completely fabricated. In the end, why do people believe what they believe? The answer is it connects with what they already believe. That's it. If you had a diet of what we call science, you'll have a foothold in a whole bunch of arenas where you can feel yourself forward, going from one little truth to another. If you are a blank slate with a bit of quackery seeded onto it, you end up believing the stars predict your life and that you can communicate with dead people with a Ouija board. CGPT doesn't have an "already believe". It just has a "humans on the panel give me reward" mechanism, and all it's doing is reflecting what it got rewarded for. Sometimes that's the scientific truth, sometimes it's crap. All the time it's confident, because that's what's rewarded.
    - petra 4y ago
      This is not a new phenomena. Have most people invested time in making sure their web search results are accurate? It is dangerous for sure, but that's what you got.
- Vvector 4y ago
  > It’s completely unreliable because it sounds confident and plausible even when it’s wrong. That applies to my boss, college professors, Wikipedia, or my neighbor after a couple of beers. It's not designed to give the correct medical steps to diagnosis a brain injury.
  - rnk 4y ago
    The problem is most people do believe these confident facts spoken by a leader, they aren't evaluating them. An easy example is the fairly widespread belief that somehow the us pres election was stolen. People said it, but there mountains of evidence don't exist. But millions of people believe it.
- williamcotton 4y ago
  You're not thinking about this probabilistically. This isn't about "correct" and "incorrect". There is an entire universe between those two poles with regards to most kinds of knowledge. Facts are a specific kind of knowledge, and frankly, not that important. It's more important to know why the Civil War was fought than to know the exact dates and locations of each battle. I've heard people say they wouldn't use ChatGPT even if there was "only a 1 in a billion chance that it made their bank account details public"... May I introduce you to some very real probabilities: https://oddsofdying.com https://oddsofdying.com If you are living in America there is a 1 in 615 chance that your cause of death will be the result of an automobile accident. So yes, it is unlikely that we will ever create a tool that can answer with 100% confidence. It is also unlikely that a manufacturing process will result in a widget that conforms to allowed tolerances 100% of the time. However, in manufacturing this is understood. A defect of 3.4 per million widgets is considered an incredibly high process capability. These tools are being made more reliable everyday. Please have a realistic goal in mind. Edit: Well I've learned this much. Many of you are not Beyesians!
  - sarchertech 4y ago
    I asked it 3 questions about distributed systems, 2 of the answers were so fundamentally wrong that they would lead to a complete misunderstanding of the subjects, and if you tried to implement systems based on that understanding they would be broken in hard to detect ways. The problem is that an LLM can only be as reliable as its training material. For example I wrote a blog post on the 2 generals problem. In the comments there is more text from inexperienced people asserting that a solution exists than there is text from my original article. An LLM trained on that article and comments will always be wrong. LLMs can be trained on smaller vetted training sets sure, but they also currently require massive amounts of data and there’s no guarantee that just waiting a few months or years will improve reliability enough to fix these issues.
    - williamcotton 4y ago
      That's a very small sample size. Also, the scope of that question is rather broad. There are a number of techniques you could try to improve your results. But I get the distinct feeling that you are not interested in improving your results as you have a motivation to "prove" something about LLMs.
- Loquebantur 4y ago
  Weirdly enough, people seem to view this only from one side: "ChatGPT is an expert at lying!". Actually, it was explicitly instructed to sound confident, serious and expertly. The important thing here is: to most people, it is enough to sound like that in order to make them believe. This heuristic is trivial to hack, gets exploited not just by ChatBots and is downright dangerous on a societal level. Acquiring and vetting information is a task of crucial importance situated in between journalism and science. Just trawling the internet and autocompleting the acquired driftwood is a weird approach to begin with. Doing so without transparency in private companies is simply folly.
  - panarky 4y ago
    I know flesh and blood humans who are confident in their knowledge but still manage to make shit up and get facts wrong. I make up my own mind about what's true and what's useful by considering an ensemble of sources. In my experience with ChatGPT, it gets less wrong and more right than most humans, and it's able to find relationships among facts and concepts that are impossible with Wikipedia and Google. It's extremely useful and powerful in its current form, in spite of its many limitations. With other tools, we don't demand perfection to find them useful. Just because a hammer sometimes misses the nail and puts a hole in the wall, just because Wikipedia sometimes gets vandalized with lies, doesn't mean hammers and Wikipedia aren't essential tools.
    - Loquebantur 4y ago
      A good hammer is designed with an optimal weight distribution that makes it easy to hit the nail. ChatGPT may be better than using your own head at times, people will still sue sooner rather than later over lost digits. However, it provides an excellent opportunity for us mere humans to reflect upon the way we organize our society in regard to vital information processing. We are awfully bad at it. So bad actually, we can't even tell how it should be in the first place and are disturbed already by some piece of software telling nonsense.
    - shrimp_emoji 4y ago
      >I know flesh and blood humans who are confident in their knowledge but still manage to make shit up and get facts wrong. I make up my own mind about what's true and what's useful by considering an ensemble of sources. They're turning the freakin' frogs gay!!!
- slackdog 4y ago
  What if instead of asking ChatGPT for facts, we instead asked it for plausible methods of determining those facts? So instead of answering "The statue of Liberty is 151 feet tall" it could instead respond with "Look it up yourself on wikipedia."
  - link_108 4y ago
    ChatGPT has made up sources when I've asked for them, so I wouldn't 100% trust it to provide great sources. Also, if ChatGPT just redirects you to search, doesn't that remove most of the value?
    - slackdog 4y ago
      > Also, if ChatGPT just redirects you to search, doesn't that remove most of the value? That's pretty much the conclusion I've already come to. I have to verify everything ChatGPT tells me, so using it is pointless if I already know where/how to look something up.
- rnk 4y ago
  It's basically like political candidate Kari Lake in Arizona. She is an extremely confident speaker, talking about the conspiracies against her, how she really won the election for Az Gov, all of them have no evidence or support. She is such an excellent speaker, she's just chilling to me because people will often just believe someone who is so well spoken. Chatgpt is that. Imagine connecting what chatgtp says with with a "super confident fascist human avatar" that is kind of Kari Lake. It will happen because it's obvious. We are just going to have ever more persuasive unreliable sources of information.
- deleted 4y ago
  [deleted]
politician 4y ago
It's fair to call it untrustworthy, however. In other words, it's not safe to trust or believe anything it says without additional validation. My policy is that teams should use ChatGPT but that everyone is individually responsible for their tasks. In other words, it's a tool, not a replacement, and if the tool does something wrong the responsibility still resides with the employee to the degree that they could have validated the results, but failed to do so. I think this strikes a good balance that preserves human jobs for as long as possible.
phailhaus 4y ago
> Dismissing it as completely unreliable is unnecessary hyperbole No it is not. If it hallucinates facts in 5% of cases, that is completely unreliable. You're basically saying "yes, it produces facts, as long as you already know the fact and double check it!" You cannot trust it with knowledge-based questions that you don't already know. I just asked it the height of the Statue of Liberty. I had to then look it up for myself just to see if it was telling the truth, because there's no citation or 'confidence level'. How is that useful?
- vidarh 4y ago
  If it only hallucinated facts in 5% of cases, it'd be more trustworthy than most people I know.
  - forgetfulness 4y ago
    You don't consult most people you know on things you'd consult with an specialist, but asking ChatGPT is portrayed as something closer to consulting with an specialist than to consulting someone on the street.
- robocat 4y ago
  Yep. Ask it a popular question, and it might get it right. Ask an equivalent question that is unknown and it gets it hilariously wrong. > What is the volume of the Statue of Liberty? The Statue of Liberty is a hollow copper statue, and its volume can be calculated by multiplying its height, width, and depth. The statue's height is 151 feet (46 meters) from the base to the torch, and its width is 35 feet (10.7 meters) at the waist. The depth of the statue, or the thickness of the copper shell, is about 2.25 inches (5.7 centimeters) throughout most of the statue. Using these measurements, the volume of the Statue of Liberty can be calculated as follows: Volume = Height x Width x Depth Volume = 151 ft x 35 ft x 0.1875 ft (2.25 inches converted to feet) Volume = 985.3125 cubic feet Therefore, the volume of the Statue of Liberty is approximately 985.3125 cubic feet (or 27.86 cubic meters). It sounds confident, the maths looks correct, but the answer is entirely wrong in multiple ways. It might be interesting to see what prompt you would need to use for it to calculate say the cylindrical volume of the main body?
pedrosbmartins 4y ago
> This is overstated and easily disproved. ChatGPT produces accurate facts as a matter of course. test it right now "the idea that this stopped clock is broken is overstated and easily disproved. the clock produces accurate time as a matter of course. go ahead and ask what time is it, just make sure it is 3:45am or 3:45pm"
- williamcotton 4y ago
  And I’m also guessing that you don’t do anything that has 1 in a billion odds of death because it’s not worth the risk, right? Edit: woooooosh...
  - pedrosbmartins 4y ago
    what? the argument here is that ChatGPT giving factual answers is a mere coincidence, not at all what the model was trained to do. It's a broken clock, it can tell you the correct time at very specific contexts, but you shouldn't rely on it as your source of factual information. If you feed him enough data saying the statue of liberty is 1 cm tall, it will happily answer a query with that "fact".
    - williamcotton 4y ago
      You're not talking about probabilities. You're talking in binaries. The broken clock is not the correct analogy.
      - remexre 4y ago
        The probability that the broken clock is right is straightforwardly 2/1440 = 0.001 = 0.1%, innit?
        williamcotton 4y ago
        To the second? To the millisecond? What are we wanting here? You're missing the point. But I'll play this silly game: ChatGPT is not incorrect 99.9% of the time.
goatlover 4y ago
When I query ChatGPT about a book or show I know well, it will start giving wrong answers after a few questions. I’ve had tell me that I was wrong when trying to correct it. I’ve also had ChatGPT generate syntactically invalid code that it provided results for. And valid code that produced different outputs from its explanation when I ran it. And then there are times when I’m having it generate fiction, and it gives me a nonsensical scenario. It’s also mistaken me for an LLM on one occasion.
tshaddox 4y ago
A lot of fictional books will contain claims that are true in the real world. It doesn’t fundamentally change the nature of what the author is doing, or whether you should trust any claim chosen at random from a fictional book.
p_j_w 4y ago
>test it right now: "How tall is the Statue of Liberty?", "In what state was Abraham Lincoln born?", etc. There are an infinite list of factual questions for which it produces accurate and correct answers. Those are easy because you can verify them. But you actually did go back and check to make sure they were right after you got answers out of it, right? And why is that, don't you trust it to just be right?
dvt 4y ago
This isn't new, but after a bit of prompt hacking, here's ChatGPT very confidently telling you the wrong height of the Statue of Liberty[1]. I made it off by 20 feet, so there's no way anyone would seriously know. [1] https://imgur.com/a/YcJ6Uu0 https://imgur.com/a/YcJ6Uu0
arp242 4y ago
To riff off the statue of liberty, a conversation (edited or brevity, to remove the ChatGPT waffling): How tall is the Statue of Liberty? The Statue of Liberty is approximately 305 feet or 93 meters tall from the ground to the tip of the torch. When did terrorists attack the statue of liberty? The Statue of Liberty has not been attacked by terrorists. However, there have been some reports and rumors circulating on the internet about such attacks. In which works of fiction did terrorists attack the statue of liberty? I am not aware of any works of fiction in which terrorists attack the Statue of Liberty. While the Statue of Liberty has been featured in many works of fiction, including movies, books, and TV shows, I cannot recall any specific instance of it being attacked by terrorists. Do you know about the game Deus Ex? Yes, Deus Ex is a popular action role-playing video game series developed by Ion Storm and published by Square Enix. The game takes place in various locations around the world, including New York City and the Statue of Liberty. In the game, the player must thwart a terrorist attack on the Statue of Liberty by infiltrating the terrorist organization and gathering intelligence. (end of chat) It directly contradicts itself here. The summary for Dues Ex is also wrong (terrorists attacked the statue of liberty before the game, and only the very first part of the game features the statue and it's not to "thwart a terrorist attack on the Statue of Liberty"). It can generate very accurate things, and can generates nonsense. However, you're never quite what it is, and there is no "well, I think it was something like this or that, but I don't know for sure". So the only way to know for sure it's accurate is to verify it with another source. And if you need to verify the accuracy then it is actually just inaccurate. Of course, humans can also generate nonsense, but you shouldn't measure ChatGPT vs. all of humanity; because it's one vs. the worst of several billion. That's not a fair match. Compared to a reasonably person ChatGPT is horribly inaccurate.
- Dylan16807 4y ago
  > It directly contradicts itself here. The summary for Dues Ex is also wrong (terrorists attacked the statue of liberty before the game, and only the very first part of the game features the statue and it's not to "thwart a terrorist attack on the Statue of Liberty"). I wouldn't call that a contradiction. I think it's fair to only count attacks that go through.
John23832 4y ago
You’re missing the entire point the op was making. The fact that these queries are correct are not because that is the intent of chatgpt’s “query engine” it because it just so happens to have been fed data that synthesizes to something sane this time (vs confident crap any other time). Compare this to a database where queries are realized against an actual corpus of data to give actual answers. The purpose of Chatgpt is to provide fluent gibberish when given a prompt. It’s a LARGE LANGUAGE MODEL. It’s only able to give you response that looks good for a language (over a distribution or words), not anything actually knowledgeable.

sarchertech 4y ago

It’s completely unreliable because it sounds confident and plausible even when it’s wrong. Essentially it fails in ways that are very hard to detect. If you ask it question you have no way to know whether it’s right or not unless you already know the answer. I’ve asked it many questions about my area of expertise, distributed systems, and it was often flat out wrong, but to a non expert it sounded perfectly plausible. I asked my my wife, a physician, to try it out and she reported the exact same problem. Some of the answers it gave had even veered off from just annoying to actually dangerous. That doesn’t mean it can’t be useful, but using it to answer important questions or using it to teach yourself something you don’t already know is dangerous.

beefield 4y ago
> It’s completely unreliable because it sounds confident and plausible even when it’s wrong This describes pretty accurately a non-trivial amount of people I have worked with.
- sarchertech 4y ago
  And hopefully after someone shows you that they can be described that way, you stop trusting what they say (at least about the thing they are known to be unreliable about).
  - vidarh 4y ago
    A whole lot of people can be trusted within a reasonable identified set of constraint, and we do, while discounting them in others where they're known to spew bullshit. It's very rarely all or nothing.
    - sarchertech 4y ago
      Obviously. Chatgpt will randomly spew bullshit about nearly any topic though. So you can really only trust it for things you are already an expert in or things that are very easy to verify.
      - williamcotton 4y ago
        It's not random. It's probabilistic. There is a very big difference. The types of errors are predictable which means they can be improved, and empirically so if you follow the academic literature.
        sarchertech 4y ago
        I have read enough of the academic literature to say with some confidence that the majority of the errors I’m talking about are not probabilistic in any meaningful sense. At least not in a way that can be predicted from the amount of readily available training data.
        zmgsabst 4y ago
        Ironically, this reply is about as useful as if chatGPT wrote it: - there’s no cited facts - it boldly proclaims its conclusion - which you’d only be able to verify as an expert …so I’m having trouble understanding what you’re complaining about with chatGPT, when that seems to be the standard for discourse.
- peoplearepeople 4y ago
  You shouldn't listen to them either then
- discreteevent 4y ago
  It may describe a number of people you have worked with but it does not describe the average medical textbook which will usually describe the best knowledge we have of a condition. If chatgpt produces something that looks like it came from a medical textbook then it is hard to blame many people for believing it. More generally, people started to trust computers more than people as soon as calculators demonstrated their reliability.
  - lordnacho 4y ago
    I don't know, I get this feeling ChatGPT has also read all the quack books where the condition depends on the alignment of stars when you were born, or how chicken entrails land when the shaman "studies" it. Those books are also written confidently without giving any sign of being completely fabricated. In the end, why do people believe what they believe? The answer is it connects with what they already believe. That's it. If you had a diet of what we call science, you'll have a foothold in a whole bunch of arenas where you can feel yourself forward, going from one little truth to another. If you are a blank slate with a bit of quackery seeded onto it, you end up believing the stars predict your life and that you can communicate with dead people with a Ouija board. CGPT doesn't have an "already believe". It just has a "humans on the panel give me reward" mechanism, and all it's doing is reflecting what it got rewarded for. Sometimes that's the scientific truth, sometimes it's crap. All the time it's confident, because that's what's rewarded.
  - petra 4y ago
    This is not a new phenomena. Have most people invested time in making sure their web search results are accurate? It is dangerous for sure, but that's what you got.
Vvector 4y ago
> It’s completely unreliable because it sounds confident and plausible even when it’s wrong. That applies to my boss, college professors, Wikipedia, or my neighbor after a couple of beers. It's not designed to give the correct medical steps to diagnosis a brain injury.
- rnk 4y ago
  The problem is most people do believe these confident facts spoken by a leader, they aren't evaluating them. An easy example is the fairly widespread belief that somehow the us pres election was stolen. People said it, but there mountains of evidence don't exist. But millions of people believe it.
williamcotton 4y ago
You're not thinking about this probabilistically. This isn't about "correct" and "incorrect". There is an entire universe between those two poles with regards to most kinds of knowledge. Facts are a specific kind of knowledge, and frankly, not that important. It's more important to know why the Civil War was fought than to know the exact dates and locations of each battle. I've heard people say they wouldn't use ChatGPT even if there was "only a 1 in a billion chance that it made their bank account details public"... May I introduce you to some very real probabilities: https://oddsofdying.com https://oddsofdying.com If you are living in America there is a 1 in 615 chance that your cause of death will be the result of an automobile accident. So yes, it is unlikely that we will ever create a tool that can answer with 100% confidence. It is also unlikely that a manufacturing process will result in a widget that conforms to allowed tolerances 100% of the time. However, in manufacturing this is understood. A defect of 3.4 per million widgets is considered an incredibly high process capability. These tools are being made more reliable everyday. Please have a realistic goal in mind. Edit: Well I've learned this much. Many of you are not Beyesians!
- sarchertech 4y ago
  I asked it 3 questions about distributed systems, 2 of the answers were so fundamentally wrong that they would lead to a complete misunderstanding of the subjects, and if you tried to implement systems based on that understanding they would be broken in hard to detect ways. The problem is that an LLM can only be as reliable as its training material. For example I wrote a blog post on the 2 generals problem. In the comments there is more text from inexperienced people asserting that a solution exists than there is text from my original article. An LLM trained on that article and comments will always be wrong. LLMs can be trained on smaller vetted training sets sure, but they also currently require massive amounts of data and there’s no guarantee that just waiting a few months or years will improve reliability enough to fix these issues.
  - williamcotton 4y ago
    That's a very small sample size. Also, the scope of that question is rather broad. There are a number of techniques you could try to improve your results. But I get the distinct feeling that you are not interested in improving your results as you have a motivation to "prove" something about LLMs.
Loquebantur 4y ago
Weirdly enough, people seem to view this only from one side: "ChatGPT is an expert at lying!". Actually, it was explicitly instructed to sound confident, serious and expertly. The important thing here is: to most people, it is enough to sound like that in order to make them believe. This heuristic is trivial to hack, gets exploited not just by ChatBots and is downright dangerous on a societal level. Acquiring and vetting information is a task of crucial importance situated in between journalism and science. Just trawling the internet and autocompleting the acquired driftwood is a weird approach to begin with. Doing so without transparency in private companies is simply folly.
- panarky 4y ago
  I know flesh and blood humans who are confident in their knowledge but still manage to make shit up and get facts wrong. I make up my own mind about what's true and what's useful by considering an ensemble of sources. In my experience with ChatGPT, it gets less wrong and more right than most humans, and it's able to find relationships among facts and concepts that are impossible with Wikipedia and Google. It's extremely useful and powerful in its current form, in spite of its many limitations. With other tools, we don't demand perfection to find them useful. Just because a hammer sometimes misses the nail and puts a hole in the wall, just because Wikipedia sometimes gets vandalized with lies, doesn't mean hammers and Wikipedia aren't essential tools.
  - Loquebantur 4y ago
    A good hammer is designed with an optimal weight distribution that makes it easy to hit the nail. ChatGPT may be better than using your own head at times, people will still sue sooner rather than later over lost digits. However, it provides an excellent opportunity for us mere humans to reflect upon the way we organize our society in regard to vital information processing. We are awfully bad at it. So bad actually, we can't even tell how it should be in the first place and are disturbed already by some piece of software telling nonsense.
  - shrimp_emoji 4y ago
    >I know flesh and blood humans who are confident in their knowledge but still manage to make shit up and get facts wrong. I make up my own mind about what's true and what's useful by considering an ensemble of sources. They're turning the freakin' frogs gay!!!
slackdog 4y ago
What if instead of asking ChatGPT for facts, we instead asked it for plausible methods of determining those facts? So instead of answering "The statue of Liberty is 151 feet tall" it could instead respond with "Look it up yourself on wikipedia."
- link_108 4y ago
  ChatGPT has made up sources when I've asked for them, so I wouldn't 100% trust it to provide great sources. Also, if ChatGPT just redirects you to search, doesn't that remove most of the value?
  - slackdog 4y ago
    > Also, if ChatGPT just redirects you to search, doesn't that remove most of the value? That's pretty much the conclusion I've already come to. I have to verify everything ChatGPT tells me, so using it is pointless if I already know where/how to look something up.
rnk 4y ago
It's basically like political candidate Kari Lake in Arizona. She is an extremely confident speaker, talking about the conspiracies against her, how she really won the election for Az Gov, all of them have no evidence or support. She is such an excellent speaker, she's just chilling to me because people will often just believe someone who is so well spoken. Chatgpt is that. Imagine connecting what chatgtp says with with a "super confident fascist human avatar" that is kind of Kari Lake. It will happen because it's obvious. We are just going to have ever more persuasive unreliable sources of information.
deleted 4y ago
[deleted]

politician 4y ago

It's fair to call it untrustworthy, however. In other words, it's not safe to trust or believe anything it says without additional validation. My policy is that teams should use ChatGPT but that everyone is individually responsible for their tasks. In other words, it's a tool, not a replacement, and if the tool does something wrong the responsibility still resides with the employee to the degree that they could have validated the results, but failed to do so. I think this strikes a good balance that preserves human jobs for as long as possible.

phailhaus 4y ago

> Dismissing it as completely unreliable is unnecessary hyperbole No it is not. If it hallucinates facts in 5% of cases, that is completely unreliable. You're basically saying "yes, it produces facts, as long as you already know the fact and double check it!" You cannot trust it with knowledge-based questions that you don't already know. I just asked it the height of the Statue of Liberty. I had to then look it up for myself just to see if it was telling the truth, because there's no citation or 'confidence level'. How is that useful?

vidarh 4y ago
If it only hallucinated facts in 5% of cases, it'd be more trustworthy than most people I know.
- forgetfulness 4y ago
  You don't consult most people you know on things you'd consult with an specialist, but asking ChatGPT is portrayed as something closer to consulting with an specialist than to consulting someone on the street.
robocat 4y ago
Yep. Ask it a popular question, and it might get it right. Ask an equivalent question that is unknown and it gets it hilariously wrong. > What is the volume of the Statue of Liberty? The Statue of Liberty is a hollow copper statue, and its volume can be calculated by multiplying its height, width, and depth. The statue's height is 151 feet (46 meters) from the base to the torch, and its width is 35 feet (10.7 meters) at the waist. The depth of the statue, or the thickness of the copper shell, is about 2.25 inches (5.7 centimeters) throughout most of the statue. Using these measurements, the volume of the Statue of Liberty can be calculated as follows: Volume = Height x Width x Depth Volume = 151 ft x 35 ft x 0.1875 ft (2.25 inches converted to feet) Volume = 985.3125 cubic feet Therefore, the volume of the Statue of Liberty is approximately 985.3125 cubic feet (or 27.86 cubic meters). It sounds confident, the maths looks correct, but the answer is entirely wrong in multiple ways. It might be interesting to see what prompt you would need to use for it to calculate say the cylindrical volume of the main body?

pedrosbmartins 4y ago

> This is overstated and easily disproved. ChatGPT produces accurate facts as a matter of course. test it right now "the idea that this stopped clock is broken is overstated and easily disproved. the clock produces accurate time as a matter of course. go ahead and ask what time is it, just make sure it is 3:45am or 3:45pm"

williamcotton 4y ago
And I’m also guessing that you don’t do anything that has 1 in a billion odds of death because it’s not worth the risk, right? Edit: woooooosh...
- pedrosbmartins 4y ago
  what? the argument here is that ChatGPT giving factual answers is a mere coincidence, not at all what the model was trained to do. It's a broken clock, it can tell you the correct time at very specific contexts, but you shouldn't rely on it as your source of factual information. If you feed him enough data saying the statue of liberty is 1 cm tall, it will happily answer a query with that "fact".
  - williamcotton 4y ago
    You're not talking about probabilities. You're talking in binaries. The broken clock is not the correct analogy.
    - remexre 4y ago
      The probability that the broken clock is right is straightforwardly 2/1440 = 0.001 = 0.1%, innit?
      - williamcotton 4y ago
        To the second? To the millisecond? What are we wanting here? You're missing the point. But I'll play this silly game: ChatGPT is not incorrect 99.9% of the time.

goatlover 4y ago

When I query ChatGPT about a book or show I know well, it will start giving wrong answers after a few questions. I’ve had tell me that I was wrong when trying to correct it. I’ve also had ChatGPT generate syntactically invalid code that it provided results for. And valid code that produced different outputs from its explanation when I ran it. And then there are times when I’m having it generate fiction, and it gives me a nonsensical scenario. It’s also mistaken me for an LLM on one occasion.

tshaddox 4y ago

A lot of fictional books will contain claims that are true in the real world. It doesn’t fundamentally change the nature of what the author is doing, or whether you should trust any claim chosen at random from a fictional book.

p_j_w 4y ago

>test it right now: "How tall is the Statue of Liberty?", "In what state was Abraham Lincoln born?", etc. There are an infinite list of factual questions for which it produces accurate and correct answers. Those are easy because you can verify them. But you actually did go back and check to make sure they were right after you got answers out of it, right? And why is that, don't you trust it to just be right?

dvt 4y ago

This isn't new, but after a bit of prompt hacking, here's ChatGPT very confidently telling you the wrong height of the Statue of Liberty[1]. I made it off by 20 feet, so there's no way anyone would seriously know. [1] https://imgur.com/a/YcJ6Uu0 https://imgur.com/a/YcJ6Uu0

arp242 4y ago

To riff off the statue of liberty, a conversation (edited or brevity, to remove the ChatGPT waffling): How tall is the Statue of Liberty? The Statue of Liberty is approximately 305 feet or 93 meters tall from the ground to the tip of the torch. When did terrorists attack the statue of liberty? The Statue of Liberty has not been attacked by terrorists. However, there have been some reports and rumors circulating on the internet about such attacks. In which works of fiction did terrorists attack the statue of liberty? I am not aware of any works of fiction in which terrorists attack the Statue of Liberty. While the Statue of Liberty has been featured in many works of fiction, including movies, books, and TV shows, I cannot recall any specific instance of it being attacked by terrorists. Do you know about the game Deus Ex? Yes, Deus Ex is a popular action role-playing video game series developed by Ion Storm and published by Square Enix. The game takes place in various locations around the world, including New York City and the Statue of Liberty. In the game, the player must thwart a terrorist attack on the Statue of Liberty by infiltrating the terrorist organization and gathering intelligence. (end of chat) It directly contradicts itself here. The summary for Dues Ex is also wrong (terrorists attacked the statue of liberty before the game, and only the very first part of the game features the statue and it's not to "thwart a terrorist attack on the Statue of Liberty"). It can generate very accurate things, and can generates nonsense. However, you're never quite what it is, and there is no "well, I think it was something like this or that, but I don't know for sure". So the only way to know for sure it's accurate is to verify it with another source. And if you need to verify the accuracy then it is actually just inaccurate. Of course, humans can also generate nonsense, but you shouldn't measure ChatGPT vs. all of humanity; because it's one vs. the worst of several billion. That's not a fair match. Compared to a reasonably person ChatGPT is horribly inaccurate.

Dylan16807 4y ago
> It directly contradicts itself here. The summary for Dues Ex is also wrong (terrorists attacked the statue of liberty before the game, and only the very first part of the game features the statue and it's not to "thwart a terrorist attack on the Statue of Liberty"). I wouldn't call that a contradiction. I think it's fair to only count attacks that go through.

John23832 4y ago

You’re missing the entire point the op was making. The fact that these queries are correct are not because that is the intent of chatgpt’s “query engine” it because it just so happens to have been fed data that synthesizes to something sane this time (vs confident crap any other time). Compare this to a database where queries are realized against an actual corpus of data to give actual answers. The purpose of Chatgpt is to provide fluent gibberish when given a prompt. It’s a LARGE LANGUAGE MODEL. It’s only able to give you response that looks good for a language (over a distribution or words), not anything actually knowledgeable.