3 ms·

Evaluating faithfulness and content selection of LLMs in book-length summaries

71 points by passwordoops 2y ago

1024core 2y ago
I didn't read the paper (just skimmed it), but no mention of Gemini 1.5 Pro? It is supposed to have the longest context window (1M available, 10M claimed in lab tests).
- hiddencost 2y ago
  Let the potato rest a minute. I suspect that they were preparing this for press by the time Gemini 1.5 Pro was released.
  - worldaroundthe 2y ago
    Gemini 1.5 Pro API was literary released yesterday. It took 11h+ for a person to evaluate a book (but that's if they sit down and do it all at once), it takes weeks to run something like this so yes... no gemini... yet...
simonw 2y ago
As far as I can tell, this study was almost entirely about fiction: https://github.com/mungg/FABLES/blob/main/booklist.md https://github.com/mungg/FABLES/blob/main/booklist.md lists 26 books, only one of which is classified as non-fiction. I would imagine that summaries of non-fiction books are evaluated quite differently from summaries of fiction. I've been trying to figure out what prompts they used. The https://github.com/mungg/FABLES https://github.com/mungg/FABLES GitHub repo says this: Summary -- (str) Entire book summarized by one of five models: Mixtral, GPT-3.5-Turbo, GPT-4, GPT-4-Turbo, and Claude-3-Opus, using the hierarchical merging method described in Chang et al.. With a link to https://arxiv.org/pdf/2310.00785.pdf https://arxiv.org/pdf/2310.00785.pdf - which then links to another GitHub repository, https://github.com/lilakk/BooookScore https://github.com/lilakk/BooookScore which has a bunch of prompts in https://github.com/lilakk/BooookScore/tree/main/prompts https://github.com/lilakk/BooookScore/tree/main/prompts Which makes me think that this original paper isn't evaluating LLMs so much as it's evaluating that one particular prompting technique for long summaries. Gemini Pro 1.5 has 1 million token context length, which should remove the need for weird hierarchical summary tricks. I wonder how well it would score?
- worldaroundthe 2y ago
  The issue with non-fiction is that some information may come from the parametric memory, rather than source text supplied to the model. So then the issue is whether the model can process the text and summarize the book or is it cheating? Fiction published in the last few months is your best shot, though sure depending on the nature of non-fiction this evaluation can be different (but some things remain the same, like you want it to be faithful/factual and not omit important information)
  - simonw 2y ago
    Thanks, I hadn't caught that. Now I understand why the study used fiction rather than non-fiction.
    - worldaroundthe 2y ago
      No prob! there was not enough explanation in the paper about this (or none). As for the prompts used for merging, there is now a link in the github repo for this.

71 points by passwordoops 2y ago

I didn't read the paper (just skimmed it), but no mention of Gemini 1.5 Pro? It is supposed to have the longest context window (1M available, 10M claimed in lab tests).

hiddencost 2y ago
Let the potato rest a minute. I suspect that they were preparing this for press by the time Gemini 1.5 Pro was released.
- worldaroundthe 2y ago
  Gemini 1.5 Pro API was literary released yesterday. It took 11h+ for a person to evaluate a book (but that's if they sit down and do it all at once), it takes weeks to run something like this so yes... no gemini... yet...

simonw 2y ago

As far as I can tell, this study was almost entirely about fiction: https://github.com/mungg/FABLES/blob/main/booklist.md https://github.com/mungg/FABLES/blob/main/booklist.md lists 26 books, only one of which is classified as non-fiction. I would imagine that summaries of non-fiction books are evaluated quite differently from summaries of fiction. I've been trying to figure out what prompts they used. The https://github.com/mungg/FABLES https://github.com/mungg/FABLES GitHub repo says this: Summary -- (str) Entire book summarized by one of five models: Mixtral, GPT-3.5-Turbo, GPT-4, GPT-4-Turbo, and Claude-3-Opus, using the hierarchical merging method described in Chang et al.. With a link to https://arxiv.org/pdf/2310.00785.pdf https://arxiv.org/pdf/2310.00785.pdf - which then links to another GitHub repository, https://github.com/lilakk/BooookScore https://github.com/lilakk/BooookScore which has a bunch of prompts in https://github.com/lilakk/BooookScore/tree/main/prompts https://github.com/lilakk/BooookScore/tree/main/prompts Which makes me think that this original paper isn't evaluating LLMs so much as it's evaluating that one particular prompting technique for long summaries. Gemini Pro 1.5 has 1 million token context length, which should remove the need for weird hierarchical summary tricks. I wonder how well it would score?

worldaroundthe 2y ago
The issue with non-fiction is that some information may come from the parametric memory, rather than source text supplied to the model. So then the issue is whether the model can process the text and summarize the book or is it cheating? Fiction published in the last few months is your best shot, though sure depending on the nature of non-fiction this evaluation can be different (but some things remain the same, like you want it to be faithful/factual and not omit important information)
- simonw 2y ago
  Thanks, I hadn't caught that. Now I understand why the study used fiction rather than non-fiction.
  - worldaroundthe 2y ago
    No prob! there was not enough explanation in the paper about this (or none). As for the prompts used for merging, there is now a link in the github repo for this.