2 ms·
It started off as an improvised test for longer context windows I think Anthropic/Claud did some early on but Greg Kamradt seems to have got it established as a
by jimmySixDOF 3y ago
It started off as an improvised test for longer context windows I think Anthropic/Claud did some early on but Greg Kamradt seems to have got it established as a benchmark [1]. Basically they hide a phrase (in this Gemini case it was text flashed on the screen) at various points in a document and then compare performance across LLMs by retrieval accuracy as the size of the document increases.
[1] https://x.com/GregKamradt/status/1758149501188210932 https://x.com/GregKamradt/status/1758149501188210932
- mptest 3y agoTo expand on this, it was so perfect at the haystack benchmark that they invented a new one, basically a hundred needle in the haystack benchmark. they had to invent new benchmarks for the model because it blew all existing ones out of the water. [0] [0]https://youtu.be/Cs6pe8o7XY8 https://youtu.be/Cs6pe8o7XY8