4 ms·

Sure — that’s very doable. I don’t have a summarization demo off-hand but it’s well-explored territory.

nutanc 4y ago
It is a well explored territory and no it can't. Especially in summarisation, it can sometimes fail suddenly and unexpectedly.
- goodside 4y ago
  Prompt: Summarize the following text: I don’t really know what to say. It’s taken so long to get to this point, but here we are. Through all the trials and tribulations we’ve faced, it all comes down to this. You, me, and the unmistakable facts of our situation. This is all that remains: The truth. The truth is something we can’t escape, or at least you can’t — not anymore. Because the truth is that you have left my pineapple slices out of the refrigerator, and thus I will not be able to partake in their joyous, fruitful delights. How dare you. You scoundrel. You wicked, wicked thing. Answer: And the completion given: The text is about a person's anger at someone else for leaving pineapple slices out of the fridge. It can, demonstrably, summarize text. The fact it sometimes makes mistakes for some texts doesn’t change that fact.
  - nutanc 4y ago
    The question was about movie scripts and children stories. Not about paragraphs. Sure it works ok for paragraphs(even then it fails many times) but the moment you cross the 2000 token limit, it does not work. When GPT3 was first released we did a lot of experiments on summarisation. We discussed a lot in the OpenAI slack. But no one could come up with a right prompt for summarisation. Yes, it works for toy paragraphs and is fun to show off. But I wouldn't build a summarisation startup on top of GPT3. Yet.
    - goodside 4y ago
      Do it in stages. Chop the text up into sections (using GPT-3 if you have to) and then summarize each page/section/chapter in isolation. Then concatenate the summarizations and summarize again. 2048 tokens is like >5KB of text — it’s not that limiting.
      - nutanc 4y ago
        The context is lost. We have tried all that. We tried that when there was free usage. Now it is too expensive. If you believe it can be done please do go ahead. There is a huge market out there for summarizing novels. No one has cracked it yet.
        goodside 4y ago
        The GPT-3 you had access to when it was free is quite different than what’s deployed today, and its ability to handle long-form inputs is its most apparent change. I’ve gotten it to give good summaries of ~5000 character texts but I admit I haven’t gone longer than the context length. Edit: Here’s an example summarizing 8,294 characters of Harry Potter fan-fiction: https://twitter.com/goodside/status/1561213457374011392?s=21&t=Bq5qVVyfZnUwrpUgpDhIyg https://twitter.com/goodside/status/1561213457374011392?s=21...
        goodside 4y ago
        Update: I got this working for all 14,410 characters of the first chapter of that fanfic. See reply in same link above.
        Vetch 4y ago
        Maybe if you've read the story it'd seem good but as someone who hasn't, I consider it a poor summary, especially the latter half.
        goodside 4y ago
        The goal wasn’t to get the post possible summary, which could be done with a more elaborate prompt detailing exactly how the summary should go. I was just demonstrating that it can get to the end of 14K chars of text and still remember both the task at hand and enough information to solve it.

It is a well explored territory and no it can't. Especially in summarisation, it can sometimes fail suddenly and unexpectedly.

goodside 4y ago
Prompt: Summarize the following text: I don’t really know what to say. It’s taken so long to get to this point, but here we are. Through all the trials and tribulations we’ve faced, it all comes down to this. You, me, and the unmistakable facts of our situation. This is all that remains: The truth. The truth is something we can’t escape, or at least you can’t — not anymore. Because the truth is that you have left my pineapple slices out of the refrigerator, and thus I will not be able to partake in their joyous, fruitful delights. How dare you. You scoundrel. You wicked, wicked thing. Answer: And the completion given: The text is about a person's anger at someone else for leaving pineapple slices out of the fridge. It can, demonstrably, summarize text. The fact it sometimes makes mistakes for some texts doesn’t change that fact.
- nutanc 4y ago
  The question was about movie scripts and children stories. Not about paragraphs. Sure it works ok for paragraphs(even then it fails many times) but the moment you cross the 2000 token limit, it does not work. When GPT3 was first released we did a lot of experiments on summarisation. We discussed a lot in the OpenAI slack. But no one could come up with a right prompt for summarisation. Yes, it works for toy paragraphs and is fun to show off. But I wouldn't build a summarisation startup on top of GPT3. Yet.
  - goodside 4y ago
    Do it in stages. Chop the text up into sections (using GPT-3 if you have to) and then summarize each page/section/chapter in isolation. Then concatenate the summarizations and summarize again. 2048 tokens is like >5KB of text — it’s not that limiting.
    - nutanc 4y ago
      The context is lost. We have tried all that. We tried that when there was free usage. Now it is too expensive. If you believe it can be done please do go ahead. There is a huge market out there for summarizing novels. No one has cracked it yet.
      - goodside 4y ago
        The GPT-3 you had access to when it was free is quite different than what’s deployed today, and its ability to handle long-form inputs is its most apparent change. I’ve gotten it to give good summaries of ~5000 character texts but I admit I haven’t gone longer than the context length. Edit: Here’s an example summarizing 8,294 characters of Harry Potter fan-fiction: https://twitter.com/goodside/status/1561213457374011392?s=21&t=Bq5qVVyfZnUwrpUgpDhIyg https://twitter.com/goodside/status/1561213457374011392?s=21...
        goodside 4y ago
        Update: I got this working for all 14,410 characters of the first chapter of that fanfic. See reply in same link above.
        Vetch 4y ago
        Maybe if you've read the story it'd seem good but as someone who hasn't, I consider it a poor summary, especially the latter half.
        goodside 4y ago
        The goal wasn’t to get the post possible summary, which could be done with a more elaborate prompt detailing exactly how the summary should go. I was just demonstrating that it can get to the end of 14K chars of text and still remember both the task at hand and enough information to solve it.