4 ms·
The Long Context benchmark numbers seem super impressive. 91% vs 49% for GPT 4.5 at 128k context length.
by M4v3R 2y ago
The Long Context benchmark numbers seem super impressive. 91% vs 49% for GPT 4.5 at 128k context length.
- Workaccount2 2y agoGoogle has the upperhand here because they are not dependent on nvidia for hardware. They make and uses their own AI accelerators.
- alexdzm 2y agoKeen to hear more about this benchmark. Is it representative of chat-to-document style usecases with big docs?
- sebzim4500 2y agoLooks like it's this benchmark [1]. It's certainly less artificial than most long context benchmarks (that are basically just a big lookup table) but probably not as representative as Fiction.LiveBench [2], which asks specific questions about works of fanfiction (which are typically excluded from training sets because they are basically porn). [1] https://arxiv.org/pdf/2409.12640 https://arxiv.org/pdf/2409.12640 [2] https://fiction.live/stories/Fiction-liveBench-Feb-20-2025/oQdzQvKHw8JyXbN87 https://fiction.live/stories/Fiction-liveBench-Feb-20-2025/o...
- swyx 2y ago"MRCR (multi-round coreference resolution)" for those looking for the link to Michaelangelo
- sebzim4500 2y agoUpdate: Gemini 2.5 also crushes fiction.livebench