4 ms·
As far as I can tell, this study was almost entirely about fiction: https://github.com/mungg/FABLES/blob/main/booklist.md https://github.com/mungg/FABLES/blob/m
by simonw 3y ago
As far as I can tell, this study was almost entirely about fiction: https://github.com/mungg/FABLES/blob/main/booklist.md https://github.com/mungg/FABLES/blob/main/booklist.md lists 26 books, only one of which is classified as non-fiction.
I would imagine that summaries of non-fiction books are evaluated quite differently from summaries of fiction.
I've been trying to figure out what prompts they used. The https://github.com/mungg/FABLES https://github.com/mungg/FABLES GitHub repo says this:
Summary -- (str) Entire book summarized
by one of five models: Mixtral,
GPT-3.5-Turbo, GPT-4, GPT-4-Turbo, and
Claude-3-Opus, using the hierarchical
merging method described in Chang et al..
With a link to https://arxiv.org/pdf/2310.00785.pdf https://arxiv.org/pdf/2310.00785.pdf - which then links to another GitHub repository, https://github.com/lilakk/BooookScore https://github.com/lilakk/BooookScore which has a bunch of prompts in https://github.com/lilakk/BooookScore/tree/main/prompts https://github.com/lilakk/BooookScore/tree/main/prompts
Which makes me think that this original paper isn't evaluating LLMs so much as it's evaluating that one particular prompting technique for long summaries.
Gemini Pro 1.5 has 1 million token context length, which should remove the need for weird hierarchical summary tricks. I wonder how well it would score?
- worldaroundthe 3y agoThe issue with non-fiction is that some information may come from the parametric memory, rather than source text supplied to the model. So then the issue is whether the model can process the text and summarize the book or is it cheating? Fiction published in the last few months is your best shot, though sure depending on the nature of non-fiction this evaluation can be different (but some things remain the same, like you want it to be faithful/factual and not omit important information)
- simonw 3y agoThanks, I hadn't caught that. Now I understand why the study used fiction rather than non-fiction.
- worldaroundthe 3y agoNo prob! there was not enough explanation in the paper about this (or none). As for the prompts used for merging, there is now a link in the github repo for this.