3 ms·
Has those results been reproduced elsewhere with other benchmarks than what Google seems to use? Hard to trust their own benchmarks at this point, and Im not h
by diggan 2y ago
Has those results been reproduced elsewhere with other benchmarks than what Google seems to use?
Hard to trust their own benchmarks at this point, and Im not home at the moment so cant try it myself either.
- llm_nerd 2y agoThey are testing for a very straightforward needle retrieval, as LLMs traditionally were terrible for this in longer contexts. There are some more advanced tests where it's far less impressive. Just a couple of days ago Adobe released one such test- https://github.com/adobe-research/NoLiMa https://github.com/adobe-research/NoLiMa