4 ms·
Asking "how many times does a particular thing happen in the book" is always going to be hard, because LLMs are notoriously bad at counting.
by simonw 2y ago
Asking "how many times does a particular thing happen in the book" is always going to be hard, because LLMs are notoriously bad at counting.
- mlyle 2y agoMan, -people- are bad at counting tasks like this.
- itake 2y agoI had issues with gpt4 last summer confusing which characters did what and hallucinating characters. his may of been a context/chunking issue (if that particular section doesn't name the character performing an action), but maybe its better now.
- campers 2y agoAsk it to output the occurrences as items in a JSON array, then count the array items in code
- torginus 2y agoNot only that, asking an LLM a question like: "List all the occurences where John Doe fights someone in the book. Also list who he fights and why" is a surefire way of getting LLMs to hallucinate, and also give incomplete info.