4 ms·

There is a benchmark, NovelQA, that LLMs don't dominate when it feels like they should. The benchmark is to read a novel and answer questions about it. LLMs ar

There is a benchmark, NovelQA, that LLMs don't dominate when it feels like they should. The benchmark is to read a novel and answer questions about it. LLMs are below human evaluation, as I last looked, but it doesn't get much attention. Once it is passed, I'd like to see one that is solving the mystery in a mystery book right before it's revealed. We'd need unpublished mystery novels to use for that benchmark, but I think it gets at what I think of as reasoning. https://novelqa.github.io/ https://novelqa.github.io/

CamperBob2 2y ago
Does it work on short stories, but not novels? If so, then that's just a minor question of context length that should self-resolve over time.
- adamgordonbell 2y ago
  The books fit in the current long context models, so it's not merely the context size constraint but the length is part of the issue, for sure.
meta_x_ai 2y ago
Looks like it's not updated for nearly a year and I'm guessing Gemini 2.0 Flash with 2m context will simply crush it
- adamgordonbell 2y ago
  That's true. They don't have Claude 3.5 on there either. So maybe it's not relevant anymore, but I'm not sure. If so, let's move on to the murder mysteries or more complex literary analysis.
rowanG077 2y ago
Benchmark how? Is it good if the LLM can or can't solve it?
loxias 2y ago
NovelQA is a great one! I also like GSM-Symbolic -- a benchmark based on making _symbolic templates_ of quite easy questions, and sampling them repeatedly, varying things like which proper nouns are used, what order relevant details appear, how many irrelevant details (GSM-NoOp) and where they are in the question, things like that. LLMs are far, _far_ below human on elementary problems, once you allow any variation and stop spoonfeeding perfectly phrased word problems. :) https://machinelearning.apple.com/research/gsm-symbolic https://machinelearning.apple.com/research/gsm-symbolic https://arxiv.org/pdf/2410.05229 https://arxiv.org/pdf/2410.05229 Paper came out in October, I don't think many have fully absorbed the implications. It's hard to take any of the claims of "LLMs can do reasoning!" seriously, once you understand that simply changing what names are used in a 8th grade math word problem can have dramatic impact on the accuracy.
latency-guy2 2y ago
> I'd like to see one that is solving the mystery in a mystery book right before it's revealed. I would think this is a not so good bench. Author does not write logically, they write for entertainment.
- adamgordonbell 2y ago
  So I'm thinking of something like Locked-room mystery where the idea is it's solvable, and the reader is given a chance to solve. The reason it seems like an interesting bench, is it's a puzzle presented in a long context. Its like testing if an LLm is at Sherlock Holmes level of world and motivation modelling.
usaar333 2y ago
That's an old leaderboard -- has no one checked any SOTA LLM in the last 8 months?

CamperBob2 2y ago

Does it work on short stories, but not novels? If so, then that's just a minor question of context length that should self-resolve over time.

adamgordonbell 2y ago
The books fit in the current long context models, so it's not merely the context size constraint but the length is part of the issue, for sure.

meta_x_ai 2y ago

Looks like it's not updated for nearly a year and I'm guessing Gemini 2.0 Flash with 2m context will simply crush it

adamgordonbell 2y ago
That's true. They don't have Claude 3.5 on there either. So maybe it's not relevant anymore, but I'm not sure. If so, let's move on to the murder mysteries or more complex literary analysis.

rowanG077 2y ago

Benchmark how? Is it good if the LLM can or can't solve it?

loxias 2y ago

NovelQA is a great one! I also like GSM-Symbolic -- a benchmark based on making _symbolic templates_ of quite easy questions, and sampling them repeatedly, varying things like which proper nouns are used, what order relevant details appear, how many irrelevant details (GSM-NoOp) and where they are in the question, things like that. LLMs are far, _far_ below human on elementary problems, once you allow any variation and stop spoonfeeding perfectly phrased word problems. :) https://machinelearning.apple.com/research/gsm-symbolic https://machinelearning.apple.com/research/gsm-symbolic https://arxiv.org/pdf/2410.05229 https://arxiv.org/pdf/2410.05229 Paper came out in October, I don't think many have fully absorbed the implications. It's hard to take any of the claims of "LLMs can do reasoning!" seriously, once you understand that simply changing what names are used in a 8th grade math word problem can have dramatic impact on the accuracy.

latency-guy2 2y ago

> I'd like to see one that is solving the mystery in a mystery book right before it's revealed. I would think this is a not so good bench. Author does not write logically, they write for entertainment.

adamgordonbell 2y ago
So I'm thinking of something like Locked-room mystery where the idea is it's solvable, and the reader is given a chance to solve. The reason it seems like an interesting bench, is it's a puzzle presented in a long context. Its like testing if an LLm is at Sherlock Holmes level of world and motivation modelling.

usaar333 2y ago

That's an old leaderboard -- has no one checked any SOTA LLM in the last 8 months?