5 ms·

I asked AI to summarize books in one sentence

5 points by WaitWaitWha 4y ago

pfdietz 4y ago
The Wizard of Oz: Transported to a surreal landscape, a young girl kills the first person she meets and then teams up with three strangers to kill again. (Ok, that was not from an AI, I think)
- mdp2021 4y ago
  Rick Polito https://images.gawker.com/183djg6lgt1jrjpg/c_fit,fl_progressive,q_80,w_636.jpg https://images.gawker.com/183djg6lgt1jrjpg/c_fit,fl_progress... Edit: I realize from sources that there seems to have been a viral diffusion on some channels of his name, the joke and its scan. But Polito's joke was famous before the Internet Age and spread by humour specialists.
  - pfdietz 4y ago
    I know. :) I wonder if the model would have repeated the joke, having possibly been trained on it somewhere.
leobg 4y ago
Did he really feed the entire book into the model? That would have been, first and foremost, expensive. And also tricky, because of the token count limit per completion. You’d have to feed it in in batches, generate sub-summaries for those. And then hopefully be able to get a final summary on your summaries. So I think what he really did is ask the model to associate a one sentence description with the title he provided. And that worked not because the model summarized the actual book. But because it had summaries done by people in its parameterized knowledge, and was able to spit out the most likely sentence from those. That is still impressive, of course. But it’s not summarization of a book. It wouldn’t have worked at all with a newly written book for which the model hasn’t already seen several human generated summaries before.
- freediver 4y ago
  Large models like GPT3 have already read the popular books (and reviews of them) so asking them to summarize it based on title is trivial. Feeding the entire book to make an actual summary, in real time is a whole another matter!
  - leobg 4y ago
    I doubt that the content of books is in the training data. OpenAI did not list Z-Library as one of their sources (though that would perhaps make the model more interesting than the alternative trained on Reddit and CommonCrawl). Even if a book’s content had been in the training data by chance (say, from a crawl of archive.org), the model would, I believe, not know where the book starts and where it ends. I don’t think it has that kind of attention span. Hence I think it comes down to aggregating human reviews that are in the training corpus. Which is cool for sure. But is not the same as having the model summarize the actual book. I find that distinction important, because there actually are algorithms that attempt to summarize text, and it would be easy to believe that GPT-3 would be capable of doing that out of the box, and much better, too. I have yet to see someone build this.
    - freediver 4y ago
      > Even if a book’s content had been in the training data by chance (say, from a crawl of archive.org), the model would, I believe, not know where the book starts and where it ends. Yeah, you are probably right. LLM summarization capabilities of such probably comes from reading the actual book reviews on the web. > I have yet to see someone build this. It is the area I am interested in. Check this: https://twitter.com/vladquant/status/1596855728735916033 https://twitter.com/vladquant/status/1596855728735916033
      - leobg 4y ago
        This looks great. Is it available to try? Might be a stretch, but if you cracked long form summarization without factual hallucination Elon/Twitter might be interested in that, as it is very much in line with what they’re trying to do, I believe. BTW, nice to see you again. Remember your name from the last time we talked here (that’s been a while ago).

5 points by WaitWaitWha 4y ago

The Wizard of Oz: Transported to a surreal landscape, a young girl kills the first person she meets and then teams up with three strangers to kill again. (Ok, that was not from an AI, I think)

mdp2021 4y ago
Rick Polito https://images.gawker.com/183djg6lgt1jrjpg/c_fit,fl_progressive,q_80,w_636.jpg https://images.gawker.com/183djg6lgt1jrjpg/c_fit,fl_progress... Edit: I realize from sources that there seems to have been a viral diffusion on some channels of his name, the joke and its scan. But Polito's joke was famous before the Internet Age and spread by humour specialists.
- pfdietz 4y ago
  I know. :) I wonder if the model would have repeated the joke, having possibly been trained on it somewhere.

leobg 4y ago

Did he really feed the entire book into the model? That would have been, first and foremost, expensive. And also tricky, because of the token count limit per completion. You’d have to feed it in in batches, generate sub-summaries for those. And then hopefully be able to get a final summary on your summaries. So I think what he really did is ask the model to associate a one sentence description with the title he provided. And that worked not because the model summarized the actual book. But because it had summaries done by people in its parameterized knowledge, and was able to spit out the most likely sentence from those. That is still impressive, of course. But it’s not summarization of a book. It wouldn’t have worked at all with a newly written book for which the model hasn’t already seen several human generated summaries before.

freediver 4y ago
Large models like GPT3 have already read the popular books (and reviews of them) so asking them to summarize it based on title is trivial. Feeding the entire book to make an actual summary, in real time is a whole another matter!
- leobg 4y ago
  I doubt that the content of books is in the training data. OpenAI did not list Z-Library as one of their sources (though that would perhaps make the model more interesting than the alternative trained on Reddit and CommonCrawl). Even if a book’s content had been in the training data by chance (say, from a crawl of archive.org), the model would, I believe, not know where the book starts and where it ends. I don’t think it has that kind of attention span. Hence I think it comes down to aggregating human reviews that are in the training corpus. Which is cool for sure. But is not the same as having the model summarize the actual book. I find that distinction important, because there actually are algorithms that attempt to summarize text, and it would be easy to believe that GPT-3 would be capable of doing that out of the box, and much better, too. I have yet to see someone build this.
  - freediver 4y ago
    > Even if a book’s content had been in the training data by chance (say, from a crawl of archive.org), the model would, I believe, not know where the book starts and where it ends. Yeah, you are probably right. LLM summarization capabilities of such probably comes from reading the actual book reviews on the web. > I have yet to see someone build this. It is the area I am interested in. Check this: https://twitter.com/vladquant/status/1596855728735916033 https://twitter.com/vladquant/status/1596855728735916033
    - leobg 4y ago
      This looks great. Is it available to try? Might be a stretch, but if you cracked long form summarization without factual hallucination Elon/Twitter might be interested in that, as it is very much in line with what they’re trying to do, I believe. BTW, nice to see you again. Remember your name from the last time we talked here (that’s been a while ago).