5 ms·

The playground summarizes it as this via GPT-4: Prompt: Please summarize the following book found on Project Gutenberg The Ruby of Kishmoor Response: "The

The playground summarizes it as this via GPT-4: Prompt: Please summarize the following book found on Project Gutenberg The Ruby of Kishmoor Response: "The Ruby of Kishmoor" is a short adventure story written by Howard Pyle. The narrative revolves around the life of Jonathan Rugg, a young man who is enticed by a mysterious stranger to come to the Caribbean to secure a valuable relic, the Ruby of Kishmoor. Once Jonathan reaches the Caribbean, he learns of the ruby's grave curse. Despite this, his innate curiosity and the prospect of considerable wealth drive him to risk his life and go after the highly prized gem. After facing a slew of challenges and trials, he surprisingly stumbles upon the ruby through sheer luck rather than organized search efforts. In the course of his journey, Jonathan delves into the depths of human greed and the lengths to which individuals go for materialistic gains. The story encapsulates an amalgamation of high-tide adventures, brushes with supernatural elements, noteworthy instances of bravery, and deep moral reflections, demanding readers to rethink about the real value of materialistic pursuits. Jonathan finally manages to escape with the ruby but at great personal cost. The story ends on a sobering note, making readers wonder if the adventures and trials were worth the prize, thus engaging them in a discourse about the true value of life and material pursuits.

CydeWeys 3y ago
Right, but that's useless without knowing how much (if any!) of it is actually correct. Is this completely hallucinated garbage?
- pmontra 3y ago
  How is it different from asking to me to summarize anything? I could have bought the book, or read the Wikipedia page, or listened people talking about it, or downloaded the torrent. In all those cases my summary could be right or could be wrong. If the rights holders know that I dowloaded the torrent they could sue me. In the other cases they can't. What if it turns out that OpenAI bought a copy of every book ingested be ChatGPT?
  - CydeWeys 3y ago
    > In all those cases my summary could be right or could be wrong. Well that's incredibly nihilistic. Whether the summary is correct or not matters a great deal! And if someone I knew said they read a book, even a very obscure one, and then summarized it to me, I'd have great confidence that they would get such simple facts as "who are the characters" and "what are the major plot points" correct. But ChatGPT? Who the hell knows? You can't trust a thing it says, especially about obscure topics. The summary is useless if you have to do a bunch of verification to see if any of it is even true, a problem that summaries even by moderately competent human writers don't have!
    - thwarted 3y ago
      if someone I knew said they read a book, even a very obscure one, and then summarized it to me, I'd have great confidence that they would get such simple facts as "who are the characters" and "what are the major plot points" correct. People, especially people you know, have reputations, based on history and experience that others have dealing with them. People can be known as liars, and anything they say is colored by such a reputation. Humans have language idioms for communicating about and dealing with such people too, phrases like "take anything that person says with a grain of salt". Look at how George Santos' history of lying about his own experience is being dealt with. ChatGPT can be (is?) the same, and it has a bad reputation for truth telling. And LLMs' reputation is not necessarily getting better in this regard. The problem is that many people attribute output that came from a machine to be of higher quality (on whatever axis) than output that came from a human, even a human they personally know and have experience dealing with. This is the same kind of prejudice as any other, or perhaps a more insidious prejudice.
      - CydeWeys 3y ago
        Agree 100% with all of this. LLMs have a huge reputation problem; you simply cannot trust what they say because they've been proven time and time again to hallucinate fictional answers. Until that problem is solved I'm struggling to see how they're as useful as people are claiming they are. You know what would be a fun test of integrity -- look up an obscure novel (potentially even the aforementioned one) that you know LLMs consistently hallucinate about because the details aren't in its training set, and then assign an essay about it as an academic assignment. It'll be pretty obvious who's read the book and who merely consulted an LLM because the latter will just be complete gibberish to anyone who actually knows what happens in that novel.
- jmull 3y ago
  This refutes the previous post’s claim that chatgpt-4 refuses to even try to provide a summary.
  - CydeWeys 3y ago
    The part that's interesting is whether the summary is correct, though. Of course, depending on how you prompt it, you might or might not get an outright refusal.
  - Wowfunhappy 3y ago
    Not necessarily, because the models have an element of randomness. Also, I was under the impression that ChatGPT has more "safeguards" (manifesting as a refusal to answer questions) than the raw API.
    - jmull 3y ago
      I don’t doubt the poster was telling the truth when they said they asked for a summary of the book and didn’t get one. It refutes the idea that chatgpt’s inability to provide a summary means it didn’t scan the original text: since it can provide a summary, the argument is entirely spurious.
      - ec109685 3y ago
        It's also a silly test since there is undoubtably a summary of most books someplace on the web.
        CydeWeys 3y ago
        You'd be surprised. There are an ungodly number of books published every year, and the back catalog is huge. You're over-indexing on popular books.
      - squeaky-clean 3y ago
        The summary it provides is entirely wrong except for the name of the main character.
maratc 3y ago
Judging by a quick glance over [0], the story indeed revolves around one Jonathan Rugg, but it looks like "manages to escape with the ruby" is completely false. Yet another hallucination I guess. [0] https://www.gutenberg.org/cache/epub/3687/pg3687-images.html https://www.gutenberg.org/cache/epub/3687/pg3687-images.html
- deleted 3y ago
  [deleted]
Imnimo 3y ago
This does at least correctly name the main character, but I think it's otherwise hallucinated. The plot of the story is that Jonathan Rugg is a Quaker who works as a clerk in Philadelphia. His boss sends him on a trip to Jamaica (credit for mentioning the Caribbean!). After arriving, he meets a woman who asks him to guard for her an ivory ball, and says that there are three men after her who want to steal it. By coincidence, he runs into the first man, they talk, he shows him the ball, and the man pulls a knife. In the struggle, the man is accidentally stabbed. Another man arrives, and sees the scene. Jonathan tries to explain, and shows him the orb. The man pulls a gun, and in the struggle is accidentally shot. A third man arrives, same story, they go down to the dock to dispose of the bodies and the man tries to steal the orb. In the struggle he is killed by when a plank of the dock collapses. Jonathan returns to the woman and says he has to return the orb to her because it's brought too much trouble. She says the men who died were the three after her, and reveals that the orb is actually a container, holding the ruby. She offers to give him the ruby and to marry him. He refuses, saying that he is already engaged back in Philadelphia, and doesn't want anything more to do with the ruby. He returns to Philadelphia and gets married, swearing off any more adventures. https://en.wikisource.org/wiki/Howard_Pyle%27s_Book_of_Pirates/Chapter_8 https://en.wikisource.org/wiki/Howard_Pyle%27s_Book_of_Pirat...
cthalupa 3y ago
I was able to get ChatGPT 4 to attempt to summarize this, but it's almost entirely hallucinated. >As of my knowledge cutoff in September 2021, the book "The Ruby of Kishmoor" is not a standalone title I am aware of. However, it is a short story by Howard Pyle which is part of his collection titled "Howard Pyle's Book of Pirates." Ruby of Kishmoor is not part of the Book of Pirates and is in fact a standalone title. >"The Ruby of Kishmoor" is an adventure tale centered on the protagonist, Jonathan Rugg. Rugg, an honest Quaker clothier from Philadelphia, sets out to sea to recover his lost wealth. On his journey, he is captured by pirates who force him to sign their articles and join their crew. He is not captured by pirates. It proceeds to summarize a long pirate story and says the story concludes with him becoming extremely wealthy because he escapes with the ruby and sells it. The summary it gave you also does not seem to match the plot of the book. Since the GP's point seems to be that having the contents of a book does not mean the model is capable of properly summarizing it, and thus supports the idea that being able to summarize something is not evidence of it containing the thing being summarized in it's dataset.

CydeWeys 3y ago

Right, but that's useless without knowing how much (if any!) of it is actually correct. Is this completely hallucinated garbage?

pmontra 3y ago
How is it different from asking to me to summarize anything? I could have bought the book, or read the Wikipedia page, or listened people talking about it, or downloaded the torrent. In all those cases my summary could be right or could be wrong. If the rights holders know that I dowloaded the torrent they could sue me. In the other cases they can't. What if it turns out that OpenAI bought a copy of every book ingested be ChatGPT?
- CydeWeys 3y ago
  > In all those cases my summary could be right or could be wrong. Well that's incredibly nihilistic. Whether the summary is correct or not matters a great deal! And if someone I knew said they read a book, even a very obscure one, and then summarized it to me, I'd have great confidence that they would get such simple facts as "who are the characters" and "what are the major plot points" correct. But ChatGPT? Who the hell knows? You can't trust a thing it says, especially about obscure topics. The summary is useless if you have to do a bunch of verification to see if any of it is even true, a problem that summaries even by moderately competent human writers don't have!
  - thwarted 3y ago
    if someone I knew said they read a book, even a very obscure one, and then summarized it to me, I'd have great confidence that they would get such simple facts as "who are the characters" and "what are the major plot points" correct. People, especially people you know, have reputations, based on history and experience that others have dealing with them. People can be known as liars, and anything they say is colored by such a reputation. Humans have language idioms for communicating about and dealing with such people too, phrases like "take anything that person says with a grain of salt". Look at how George Santos' history of lying about his own experience is being dealt with. ChatGPT can be (is?) the same, and it has a bad reputation for truth telling. And LLMs' reputation is not necessarily getting better in this regard. The problem is that many people attribute output that came from a machine to be of higher quality (on whatever axis) than output that came from a human, even a human they personally know and have experience dealing with. This is the same kind of prejudice as any other, or perhaps a more insidious prejudice.
    - CydeWeys 3y ago
      Agree 100% with all of this. LLMs have a huge reputation problem; you simply cannot trust what they say because they've been proven time and time again to hallucinate fictional answers. Until that problem is solved I'm struggling to see how they're as useful as people are claiming they are. You know what would be a fun test of integrity -- look up an obscure novel (potentially even the aforementioned one) that you know LLMs consistently hallucinate about because the details aren't in its training set, and then assign an essay about it as an academic assignment. It'll be pretty obvious who's read the book and who merely consulted an LLM because the latter will just be complete gibberish to anyone who actually knows what happens in that novel.
jmull 3y ago
This refutes the previous post’s claim that chatgpt-4 refuses to even try to provide a summary.
- CydeWeys 3y ago
  The part that's interesting is whether the summary is correct, though. Of course, depending on how you prompt it, you might or might not get an outright refusal.
- Wowfunhappy 3y ago
  Not necessarily, because the models have an element of randomness. Also, I was under the impression that ChatGPT has more "safeguards" (manifesting as a refusal to answer questions) than the raw API.
  - jmull 3y ago
    I don’t doubt the poster was telling the truth when they said they asked for a summary of the book and didn’t get one. It refutes the idea that chatgpt’s inability to provide a summary means it didn’t scan the original text: since it can provide a summary, the argument is entirely spurious.
    - ec109685 3y ago
      It's also a silly test since there is undoubtably a summary of most books someplace on the web.
      - CydeWeys 3y ago
        You'd be surprised. There are an ungodly number of books published every year, and the back catalog is huge. You're over-indexing on popular books.
    - squeaky-clean 3y ago
      The summary it provides is entirely wrong except for the name of the main character.

maratc 3y ago

Judging by a quick glance over [0], the story indeed revolves around one Jonathan Rugg, but it looks like "manages to escape with the ruby" is completely false. Yet another hallucination I guess. [0] https://www.gutenberg.org/cache/epub/3687/pg3687-images.html https://www.gutenberg.org/cache/epub/3687/pg3687-images.html

deleted 3y ago
[deleted]

Imnimo 3y ago

This does at least correctly name the main character, but I think it's otherwise hallucinated. The plot of the story is that Jonathan Rugg is a Quaker who works as a clerk in Philadelphia. His boss sends him on a trip to Jamaica (credit for mentioning the Caribbean!). After arriving, he meets a woman who asks him to guard for her an ivory ball, and says that there are three men after her who want to steal it. By coincidence, he runs into the first man, they talk, he shows him the ball, and the man pulls a knife. In the struggle, the man is accidentally stabbed. Another man arrives, and sees the scene. Jonathan tries to explain, and shows him the orb. The man pulls a gun, and in the struggle is accidentally shot. A third man arrives, same story, they go down to the dock to dispose of the bodies and the man tries to steal the orb. In the struggle he is killed by when a plank of the dock collapses. Jonathan returns to the woman and says he has to return the orb to her because it's brought too much trouble. She says the men who died were the three after her, and reveals that the orb is actually a container, holding the ruby. She offers to give him the ruby and to marry him. He refuses, saying that he is already engaged back in Philadelphia, and doesn't want anything more to do with the ruby. He returns to Philadelphia and gets married, swearing off any more adventures. https://en.wikisource.org/wiki/Howard_Pyle%27s_Book_of_Pirates/Chapter_8 https://en.wikisource.org/wiki/Howard_Pyle%27s_Book_of_Pirat...

cthalupa 3y ago

I was able to get ChatGPT 4 to attempt to summarize this, but it's almost entirely hallucinated. >As of my knowledge cutoff in September 2021, the book "The Ruby of Kishmoor" is not a standalone title I am aware of. However, it is a short story by Howard Pyle which is part of his collection titled "Howard Pyle's Book of Pirates." Ruby of Kishmoor is not part of the Book of Pirates and is in fact a standalone title. >"The Ruby of Kishmoor" is an adventure tale centered on the protagonist, Jonathan Rugg. Rugg, an honest Quaker clothier from Philadelphia, sets out to sea to recover his lost wealth. On his journey, he is captured by pirates who force him to sign their articles and join their crew. He is not captured by pirates. It proceeds to summarize a long pirate story and says the story concludes with him becoming extremely wealthy because he escapes with the ruby and sells it. The summary it gave you also does not seem to match the plot of the book. Since the GP's point seems to be that having the contents of a book does not mean the model is capable of properly summarizing it, and thus supports the idea that being able to summarize something is not evidence of it containing the thing being summarized in it's dataset.