5 ms·

> “If you ask it about something it doesn't know, right then and there, it will concoct a fiction for you. It won't say "I don't know,"” It doesn’t know that i

> “If you ask it about something it doesn't know, right then and there, it will concoct a fiction for you. It won't say "I don't know,"” It doesn’t know that it doesn’t know. Texts it was trained on rarely go “On the History of the Greek Language, by Andros Dimos; ahem what is the history of the Greek language? I don’t know. The end.”

A_D_E_P_T 2y ago
Hmmm... well... Let me give you an example. The other day I asked for a summary of the short story "The Palace at Midnight" by Robert Silverberg. GPT-4o responded: "The Palace at Midnight" by Robert Silverberg is a science fiction short story that was first published in 1980. It is part of Silverberg’s "Majipoor" series, but stands alone in terms of its setting and characters, distinct from the expansive world-building typical of the series. The story is set in a dystopian future in India and revolves around themes of memory, loss, and the haunting allure of the past. The narrative follows the protagonist, Edward, and his group of friends who are survivors of a nuclear war that has devastated much of the world. . . All of which was complete nonsense. "The Palace at Midnight" is a short story of a few pages that has nothing to do with Majipoor, was published in 1981, and was set in San Francisco. It doesn't know -- it knows that it doesn't know -- so it just makes something up that seems superficially credible. Because it aims to please and be helpful; altogether too helpful.
- jodrellblank 2y ago
  > “it knows that it doesn't know” No it doesn’t; it isn’t an intelligence, it’s a token generator based on patterns in the inputs it was trained on (and those are explanations and answers, not millions of people writing “I don’t know”).
  - A_D_E_P_T 2y ago
    Right, but it may be a fixable problem, even under the circumstances. These token generators have vast databases at their disposal and have crawled the entire public-facing internet. They should be able to assign confidence values to their drafts/statements, and reject low-confidence drafts/statements before putting them in front of users. They could do this by fact-checking their drafts in an adversarial way. For instance, a cursory check of isfdb.org would have given ChatGPT better information than I was provided with.

A_D_E_P_T 2y ago

Hmmm... well... Let me give you an example. The other day I asked for a summary of the short story "The Palace at Midnight" by Robert Silverberg. GPT-4o responded: "The Palace at Midnight" by Robert Silverberg is a science fiction short story that was first published in 1980. It is part of Silverberg’s "Majipoor" series, but stands alone in terms of its setting and characters, distinct from the expansive world-building typical of the series. The story is set in a dystopian future in India and revolves around themes of memory, loss, and the haunting allure of the past. The narrative follows the protagonist, Edward, and his group of friends who are survivors of a nuclear war that has devastated much of the world. . . All of which was complete nonsense. "The Palace at Midnight" is a short story of a few pages that has nothing to do with Majipoor, was published in 1981, and was set in San Francisco. It doesn't know -- it knows that it doesn't know -- so it just makes something up that seems superficially credible. Because it aims to please and be helpful; altogether too helpful.

jodrellblank 2y ago
> “it knows that it doesn't know” No it doesn’t; it isn’t an intelligence, it’s a token generator based on patterns in the inputs it was trained on (and those are explanations and answers, not millions of people writing “I don’t know”).
- A_D_E_P_T 2y ago
  Right, but it may be a fixable problem, even under the circumstances. These token generators have vast databases at their disposal and have crawled the entire public-facing internet. They should be able to assign confidence values to their drafts/statements, and reject low-confidence drafts/statements before putting them in front of users. They could do this by fact-checking their drafts in an adversarial way. For instance, a cursory check of isfdb.org would have given ChatGPT better information than I was provided with.