4 ms·
Generally this fails. Most LLMs lose the ability to track facts over about 20k words of content, the best can manage maybe 40k words. Look for "needle" benchm
by suprjami 2y ago
Generally this fails.
Most LLMs lose the ability to track facts over about 20k words of content, the best can manage maybe 40k words.
Look for "needle" benchmark tests, as in needle-in-haystack.
Not to mention the memory requirements of such a huge context like 128k or 1M tokens. Only people with enterprise servers at home could run that locally.
- wahnfrieden 2y agoWhat about scanning over chunks of data to collect matches iteratively - that’s what I meant rather than loading full context limits
- learningcircuit 2y agoVery good answer. It is very hard with small LLM.
- HashedViking 2y agoGood take on that. I still think a q8 32B model with a 200k context would fit into the 48Gb VRAM of one of those modded RTX 4090.