4 ms·

How good are these models at summarization anyways? I tried uploading obscure books I've already read, to GPT4 and Claude 3 and asked them to summarize the plot

How good are these models at summarization anyways? I tried uploading obscure books I've already read, to GPT4 and Claude 3 and asked them to summarize the plot and particular details, as well as asking how many times does a particular thing happen in the book, and the results have been hit and miss. I certainly would not trust these models to create comprehensive and correct summaries of highly sensitive records.

kainan-ai 2y ago
Yeah its more creative than other fine tunes, you'd need to make a pretty strict system prompt then test before doing anything with sensitive records
simonw 2y ago
Asking "how many times does a particular thing happen in the book" is always going to be hard, because LLMs are notoriously bad at counting.
- mlyle 2y ago
  Man, -people- are bad at counting tasks like this.
- itake 2y ago
  I had issues with gpt4 last summer confusing which characters did what and hallucinating characters. his may of been a context/chunking issue (if that particular section doesn't name the character performing an action), but maybe its better now.
  - sweca 2y ago
    Were you using the API or ChatGPT? ChatGPT uses RAG on large documents which is useless for many tasks.
  - awwaiid 2y ago
    Last summer was like a million years ago! You should try again :)
- campers 2y ago
  Ask it to output the occurrences as items in a JSON array, then count the array items in code
- torginus 2y ago
  Not only that, asking an LLM a question like: "List all the occurences where John Doe fights someone in the book. Also list who he fights and why" is a surefire way of getting LLMs to hallucinate, and also give incomplete info.
miki123211 2y ago
For whatever it's worth, asking a model to count is a terrible idea due to how they work. You may have more luck with a hybrid approach, using LLMs for language understanding and computers for the counting. For example, ask them to write a short, one-line description of every instance where something happens, and then use a traditional program to count the lines.
phren0logy 2y ago
If you chunk the document well (which is surprisingly tricky with messy documents), then you can do summary via map reduce or other techniques. To be clear: I don't trust it to provide an accurate summary; it's not meant to replace reading the documents. But it may help to find relevant parts of a document later, or get a reasonable overview of documents before starting a manual review. I expect (but will have to see) if it's better than just opening up a random PDF and starting there. I recently had a case with over 24k pages of records, some of which were in PDFs that were thousands of pages long. The ability to do semantic search rather than keyword search was useful for when I said to myself, "Didn't I see something about that before? Where was that?"
- sroussey 2y ago
  I’ve found summarizing this way is a great idea but the summarizing models are not great themselves. Would even prefer an embedding with the summary as maybe the embedding would be better.

kainan-ai 2y ago

Yeah its more creative than other fine tunes, you'd need to make a pretty strict system prompt then test before doing anything with sensitive records

simonw 2y ago

Asking "how many times does a particular thing happen in the book" is always going to be hard, because LLMs are notoriously bad at counting.

mlyle 2y ago
Man, -people- are bad at counting tasks like this.
itake 2y ago
I had issues with gpt4 last summer confusing which characters did what and hallucinating characters. his may of been a context/chunking issue (if that particular section doesn't name the character performing an action), but maybe its better now.
- sweca 2y ago
  Were you using the API or ChatGPT? ChatGPT uses RAG on large documents which is useless for many tasks.
- awwaiid 2y ago
  Last summer was like a million years ago! You should try again :)
campers 2y ago
Ask it to output the occurrences as items in a JSON array, then count the array items in code
torginus 2y ago
Not only that, asking an LLM a question like: "List all the occurences where John Doe fights someone in the book. Also list who he fights and why" is a surefire way of getting LLMs to hallucinate, and also give incomplete info.

miki123211 2y ago

For whatever it's worth, asking a model to count is a terrible idea due to how they work. You may have more luck with a hybrid approach, using LLMs for language understanding and computers for the counting. For example, ask them to write a short, one-line description of every instance where something happens, and then use a traditional program to count the lines.

phren0logy 2y ago

If you chunk the document well (which is surprisingly tricky with messy documents), then you can do summary via map reduce or other techniques. To be clear: I don't trust it to provide an accurate summary; it's not meant to replace reading the documents. But it may help to find relevant parts of a document later, or get a reasonable overview of documents before starting a manual review. I expect (but will have to see) if it's better than just opening up a random PDF and starting there. I recently had a case with over 24k pages of records, some of which were in PDFs that were thousands of pages long. The ability to do semantic search rather than keyword search was useful for when I said to myself, "Didn't I see something about that before? Where was that?"

sroussey 2y ago
I’ve found summarizing this way is a great idea but the summarizing models are not great themselves. Would even prefer an embedding with the summary as maybe the embedding would be better.