3 ms·

The great problem with ChatGPT is that it's a sycophant and aims to please. If you ask it about something it doesn't know, right then and there, it will concoc

The great problem with ChatGPT is that it's a sycophant and aims to please. If you ask it about something it doesn't know, right then and there, it will concoct a fiction for you. It won't say "I don't know," or "I can't help with that." If you coach it to respond to something in a certain way, it'll respond that way for you as its top priority. If you ask it to review a text, it'll usually find a way to give you at least a 7 or 8 out of 10. (Though, interestingly, rarely a 10/10 score. You can upload excerpts from some of the great works of literature and philosophy and see ChatGPT give them an 8/10, just as it gives an 8/10 to your essay or blog post.) Practically the only way to get a halfway critical response is to add the words "be critical" to your prompt. A more circumspect and less obsequious ChatGPT would solve a lot of problems.

zertar 2y ago
Interesting. Copilot on the other hand is more like a conceited, passive aggressive brat who uses some polite phrases but aborts conversations if you contradict it too much. The only smooth conversations with Copilot are the ones where you allow it to regurgitate "facts" and act in a submissive manner.
jodrellblank 2y ago
> “If you ask it about something it doesn't know, right then and there, it will concoct a fiction for you. It won't say "I don't know,"” It doesn’t know that it doesn’t know. Texts it was trained on rarely go “On the History of the Greek Language, by Andros Dimos; ahem what is the history of the Greek language? I don’t know. The end.”
- A_D_E_P_T 2y ago
  Hmmm... well... Let me give you an example. The other day I asked for a summary of the short story "The Palace at Midnight" by Robert Silverberg. GPT-4o responded: "The Palace at Midnight" by Robert Silverberg is a science fiction short story that was first published in 1980. It is part of Silverberg’s "Majipoor" series, but stands alone in terms of its setting and characters, distinct from the expansive world-building typical of the series. The story is set in a dystopian future in India and revolves around themes of memory, loss, and the haunting allure of the past. The narrative follows the protagonist, Edward, and his group of friends who are survivors of a nuclear war that has devastated much of the world. . . All of which was complete nonsense. "The Palace at Midnight" is a short story of a few pages that has nothing to do with Majipoor, was published in 1981, and was set in San Francisco. It doesn't know -- it knows that it doesn't know -- so it just makes something up that seems superficially credible. Because it aims to please and be helpful; altogether too helpful.
  - jodrellblank 2y ago
    > “it knows that it doesn't know” No it doesn’t; it isn’t an intelligence, it’s a token generator based on patterns in the inputs it was trained on (and those are explanations and answers, not millions of people writing “I don’t know”).
    - A_D_E_P_T 2y ago
      Right, but it may be a fixable problem, even under the circumstances. These token generators have vast databases at their disposal and have crawled the entire public-facing internet. They should be able to assign confidence values to their drafts/statements, and reject low-confidence drafts/statements before putting them in front of users. They could do this by fact-checking their drafts in an adversarial way. For instance, a cursory check of isfdb.org would have given ChatGPT better information than I was provided with.
rchaud 2y ago
ChatGPT is the search engine equivalent of those streaming sites that won't directly admit they don't have the show you explicitly searched for, but will instead show you a list of other shows they think will placate you, and pretend they're indistinguishable.

zertar 2y ago

Interesting. Copilot on the other hand is more like a conceited, passive aggressive brat who uses some polite phrases but aborts conversations if you contradict it too much. The only smooth conversations with Copilot are the ones where you allow it to regurgitate "facts" and act in a submissive manner.

jodrellblank 2y ago

> “If you ask it about something it doesn't know, right then and there, it will concoct a fiction for you. It won't say "I don't know,"” It doesn’t know that it doesn’t know. Texts it was trained on rarely go “On the History of the Greek Language, by Andros Dimos; ahem what is the history of the Greek language? I don’t know. The end.”

A_D_E_P_T 2y ago
Hmmm... well... Let me give you an example. The other day I asked for a summary of the short story "The Palace at Midnight" by Robert Silverberg. GPT-4o responded: "The Palace at Midnight" by Robert Silverberg is a science fiction short story that was first published in 1980. It is part of Silverberg’s "Majipoor" series, but stands alone in terms of its setting and characters, distinct from the expansive world-building typical of the series. The story is set in a dystopian future in India and revolves around themes of memory, loss, and the haunting allure of the past. The narrative follows the protagonist, Edward, and his group of friends who are survivors of a nuclear war that has devastated much of the world. . . All of which was complete nonsense. "The Palace at Midnight" is a short story of a few pages that has nothing to do with Majipoor, was published in 1981, and was set in San Francisco. It doesn't know -- it knows that it doesn't know -- so it just makes something up that seems superficially credible. Because it aims to please and be helpful; altogether too helpful.
- jodrellblank 2y ago
  > “it knows that it doesn't know” No it doesn’t; it isn’t an intelligence, it’s a token generator based on patterns in the inputs it was trained on (and those are explanations and answers, not millions of people writing “I don’t know”).
  - A_D_E_P_T 2y ago
    Right, but it may be a fixable problem, even under the circumstances. These token generators have vast databases at their disposal and have crawled the entire public-facing internet. They should be able to assign confidence values to their drafts/statements, and reject low-confidence drafts/statements before putting them in front of users. They could do this by fact-checking their drafts in an adversarial way. For instance, a cursory check of isfdb.org would have given ChatGPT better information than I was provided with.

rchaud 2y ago

ChatGPT is the search engine equivalent of those streaming sites that won't directly admit they don't have the show you explicitly searched for, but will instead show you a list of other shows they think will placate you, and pretend they're indistinguishable.