3 ms·
This would have been bigger news except for Gemini 1.5, Sora, and the Magic investment all happening at the same time. Gemini can do needle in a haystack reli
by jimmySixDOF 3y ago
This would have been bigger news except for Gemini 1.5, Sora, and the Magic investment all happening at the same time. Gemini can do needle in a haystack reliably in the 3hrs of video they tested against.
- cma 3y agoMagic investment?
- stormfather 3y agoMagic.dev got ~100MM investment because the ex CEO of github was super impressed by their progress towards super long context, code optimized LLMs to replace developers...
- sebastiennight 3y agoWhat is the "needle in a haystack" test you're referring to?
- jimmySixDOF 3y agoIt started off as an improvised test for longer context windows I think Anthropic/Claud did some early on but Greg Kamradt seems to have got it established as a benchmark [1]. Basically they hide a phrase (in this Gemini case it was text flashed on the screen) at various points in a document and then compare performance across LLMs by retrieval accuracy as the size of the document increases. [1] https://x.com/GregKamradt/status/1758149501188210932 https://x.com/GregKamradt/status/1758149501188210932
- mptest 3y agoTo expand on this, it was so perfect at the haystack benchmark that they invented a new one, basically a hundred needle in the haystack benchmark. they had to invent new benchmarks for the model because it blew all existing ones out of the water. [0] [0]https://youtu.be/Cs6pe8o7XY8 https://youtu.be/Cs6pe8o7XY8