5 ms·

That's exactly the wrong sort of thing to turn to ChatGPT for. In probing it for more detail, you're giving it increasingly helpful hints on how to answer your

That's exactly the wrong sort of thing to turn to ChatGPT for. In probing it for more detail, you're giving it increasingly helpful hints on how to answer your questions in a way that makes you satisfied, regardless of accuracy. That's what it's optimized to do. And because you're working from a humble and curious understanding of the topic yourself, it can and will make up all sorts of things that make you believe its insightful because you want to believe it. It's the same technique a stage psychic uses to demonstrate that they knew all about your late uncle and his peculiar love of ducks.

darkwater 4y ago
GP stated that they asked ChatGPT about a topic they already were familiar with, from other sources. And that the answer made sense.
- Retric 4y ago
  ChatGPT is optimized to seem to make sense, it’s like talking to a world class con artist you won’t notice it’s talking nonsense. Thus is subtly dangerous in ways that aren’t obvious in the beginning. Sure it seems great at writing boilerplate code, unfortunately it’s really good and writing the kind of bugs that you don’t notice. This is maddening in a completely novel way, like working with an actively malicious autocomplete.
  - 2devnull 4y ago
    You can verify it’s accuracy. The metaphor of “con artist” is a poor one. It has no adversarial interest. A con artist does. Furthermore, con stands for confidence. A con artist manipulates confidence. Gpt and other language models are “what it says on the tin.” Not understanding the fallibility of LLM is not the fault of the technology. It’s the user’s fault. Not at all like a con artist. More like a tool that can be dangerous in the hands of the uneducated. That is a very large set of human tools, from nuclear reaction to pharmaceuticals to the automobile. Perhaps we just need “drivers ed” for large language models.
    - Retric 4y ago
      The problem is needing to verify accuracy makes it a really bad search engine replacement. A good con artist is generally truthful, they just know how to seem truthful even when they lie which exactly the issue with trusting ChatGPT etc. You can’t tell if the model is telling bullshit unless you already know the truth. This doesn’t make them useless in general, just a poor fit if you can’t easily fact check the results.
      - 2devnull 4y ago
        Good point. I don’t think they will be a good search engine replacement, but maybe a powerful supplement for some users. And honestly with SEO and other issues even search engines can be manipulated in ways that an average user might have difficulty detecting.
      - je42 4y ago
        Easy, you use a second source and double check. In google you would go to another search result. For ChatGPT you actually need to verify via Google or actually try it when it is code.
        Retric 4y ago
        What’s the point of using ChatGPT when searching if you still need to verify the information through another search?
- huimang 4y ago
  Conversely I asked it some simple questions about some songs' harmonic structures, and music theory, and it got things quite wrong. There's already an example of it in the article where it incorrectly stated the tritone of F was Bb.
yellow_postit 4y ago
To OP’s point what’s with the gate keeping on “right” or “wrong” ways to use it? It’s a tool. It has flaws. It can be useful and if it is, have at it.
- Retric 4y ago
  The issue is that approach maximizes its flaws. It’s like shooing yourself in the foot with a nail gun, that’s objectively the wrong thing to do.
- b3morales 4y ago
  If there was a hammer that, held one way, drove nails perfectly in one blow, and held another, made it look like the nail was driven but actually broke it right at the junction so that the work pieces weren't fastened... I'd say that the second way was the wrong way to use that hammer.
  - efitz 4y ago
    Not only wrong, but dangerous, because nails are often used to fasten structural elements of houses, and incorrect but hard-to-detect flaws like this could result in collapse. Similarly, if ChatGPT gives you an answer high in truthiness but low in accuracy, it could negatively impact you, whether loss in credibility if you repeat nonsense in front of someone knowledgable, or even worse if you use the incorrect knowledge to try to solve a real world problem.
- naniwaduni 4y ago
  It's a tool whose flaw is that sometimes it secretly does the exact opposite of what you're trying to do with it in a way that's only detectable if you didn't already need it. It's like Babelfish, but for monoglots.
hansvm 4y ago
If you're able to validate the results though (so they logically fit together, or with that newfound vocabulary and pseudounderstanding can you search for that information elsewhere), is that really such a bad thing? I've been doing exactly that to quickly ramp up on some new numerical algorithms I previously didn't know existed.
- naniwaduni 4y ago
  That's a really big if.
  - hansvm 4y ago
    Why? It's the same standard you need to apply to anything else.

darkwater 4y ago

GP stated that they asked ChatGPT about a topic they already were familiar with, from other sources. And that the answer made sense.

Retric 4y ago
ChatGPT is optimized to seem to make sense, it’s like talking to a world class con artist you won’t notice it’s talking nonsense. Thus is subtly dangerous in ways that aren’t obvious in the beginning. Sure it seems great at writing boilerplate code, unfortunately it’s really good and writing the kind of bugs that you don’t notice. This is maddening in a completely novel way, like working with an actively malicious autocomplete.
- 2devnull 4y ago
  You can verify it’s accuracy. The metaphor of “con artist” is a poor one. It has no adversarial interest. A con artist does. Furthermore, con stands for confidence. A con artist manipulates confidence. Gpt and other language models are “what it says on the tin.” Not understanding the fallibility of LLM is not the fault of the technology. It’s the user’s fault. Not at all like a con artist. More like a tool that can be dangerous in the hands of the uneducated. That is a very large set of human tools, from nuclear reaction to pharmaceuticals to the automobile. Perhaps we just need “drivers ed” for large language models.
  - Retric 4y ago
    The problem is needing to verify accuracy makes it a really bad search engine replacement. A good con artist is generally truthful, they just know how to seem truthful even when they lie which exactly the issue with trusting ChatGPT etc. You can’t tell if the model is telling bullshit unless you already know the truth. This doesn’t make them useless in general, just a poor fit if you can’t easily fact check the results.
    - 2devnull 4y ago
      Good point. I don’t think they will be a good search engine replacement, but maybe a powerful supplement for some users. And honestly with SEO and other issues even search engines can be manipulated in ways that an average user might have difficulty detecting.
    - je42 4y ago
      Easy, you use a second source and double check. In google you would go to another search result. For ChatGPT you actually need to verify via Google or actually try it when it is code.
      - Retric 4y ago
        What’s the point of using ChatGPT when searching if you still need to verify the information through another search?
huimang 4y ago
Conversely I asked it some simple questions about some songs' harmonic structures, and music theory, and it got things quite wrong. There's already an example of it in the article where it incorrectly stated the tritone of F was Bb.

yellow_postit 4y ago

To OP’s point what’s with the gate keeping on “right” or “wrong” ways to use it? It’s a tool. It has flaws. It can be useful and if it is, have at it.

Retric 4y ago
The issue is that approach maximizes its flaws. It’s like shooing yourself in the foot with a nail gun, that’s objectively the wrong thing to do.
b3morales 4y ago
If there was a hammer that, held one way, drove nails perfectly in one blow, and held another, made it look like the nail was driven but actually broke it right at the junction so that the work pieces weren't fastened... I'd say that the second way was the wrong way to use that hammer.
- efitz 4y ago
  Not only wrong, but dangerous, because nails are often used to fasten structural elements of houses, and incorrect but hard-to-detect flaws like this could result in collapse. Similarly, if ChatGPT gives you an answer high in truthiness but low in accuracy, it could negatively impact you, whether loss in credibility if you repeat nonsense in front of someone knowledgable, or even worse if you use the incorrect knowledge to try to solve a real world problem.
naniwaduni 4y ago
It's a tool whose flaw is that sometimes it secretly does the exact opposite of what you're trying to do with it in a way that's only detectable if you didn't already need it. It's like Babelfish, but for monoglots.

hansvm 4y ago

If you're able to validate the results though (so they logically fit together, or with that newfound vocabulary and pseudounderstanding can you search for that information elsewhere), is that really such a bad thing? I've been doing exactly that to quickly ramp up on some new numerical algorithms I previously didn't know existed.

naniwaduni 4y ago
That's a really big if.
- hansvm 4y ago
  Why? It's the same standard you need to apply to anything else.