4 ms·
I ran a couple needle-in-a-haystack type queries with just a 32k context length, and was very much not impressed. It often failed to find facts buried in the mi
by lukev 2y ago
I ran a couple needle-in-a-haystack type queries with just a 32k context length, and was very much not impressed. It often failed to find facts buried in the middle of the prompt, that were stated almost identically to the question being asked.
It's cool that these models are getting such long contexts, but performance definitely degrades the longer the context gets and I haven't seen this characterized or quantified very well anywhere.
- zackangelo 2y agoWould you care to share your prompts? They posted a haystack benchmark in the blog post that seems too good to be true.
- busssard 2y agoyeah, when i saw that they have 100% coverage with 1M token, i thought this must be a placeholder image, for when the actual results come in. Because there is no variation, nothing.
- lukev 2y agoI wasn't scientific about it, unfortunately. My searches were natural language, not token-based, though.