3 ms·
I've been testing AI as a beta reader for >100k novels, and I can tell you with 100% certainty that Claude gets confused about things across long contexts much
by CuriouslyC 1y ago
I've been testing AI as a beta reader for >100k novels, and I can tell you with 100% certainty that Claude gets confused about things across long contexts much sooner than either O3/GPT5 or Gemini 2.5. In my experience Gemini 2.5 and O3/GPT5 run neck and neck until around 80-100k tokens, then Gemini 2.5 starts to pull ahead and by 150k tokens it's absolutely dominant. Claude is respectable but clearly in third place.
https://fiction.live/stories/Fiction-liveBench-Mar-25-2025/oQdzQvKHw8JyXbN87 https://fiction.live/stories/Fiction-liveBench-Mar-25-2025/o...
https://longbench2.github.io/ https://longbench2.github.io/
- itsafarqueue 1y agoReally useful comment thanks. Reminder that LLMs aren’t just for coding.