4 ms·
> * Huge context window But how well does it actually handle that context window? E.g. a lot of models support 200K context, but the LLM can only really work w
by ai-christianson 2y ago
> * Huge context window
But how well does it actually handle that context window? E.g. a lot of models support 200K context, but the LLM can only really work with ~80K or so of it before it starts to get confused.
- summerlight 2y agoMy experience is that Gemini works relatively well on larger contexts. Not perfect, but more reliable.
- Havoc 2y agoI'm sure someone will do a haystack test, but from my casual testing it seems pretty good
- asadm 2y agoit works REALLY well. I have used it to dump many references codes and then help me write a new modules etc. I have gone up to 200k tokens I think with no problems in recall.
- ai-christianson 2y agoAwesome. Models that can usefully leverage such large context windows are rare at this point. Something like this opens up a lot of use cases.
- f38zf5vdt 2y agoIt works okay out to roughly 20-40k tokens. Once the window gets larger than that, it degrades significantly. You can needle in the haystack out to that distance, but asking it for multiple things from the document leads to hallucinations for me. Ironic, but GPT4o works better for me at longer contexts <128k than Gemini 2.0 flash. And out to 1m is just hopeless, even though you can do it.
- llm_nerd 2y agoThere is the needle in the haystack measure which is, as you probably guessed, hiding a small fact in a massive set of tokens and asking it to recall it. Recent Gemini models actually do extraordinarily well. https://cloud.google.com/blog/products/ai-machine-learning/the-needle-in-the-haystack-test-and-how-gemini-pro-solves-it https://cloud.google.com/blog/products/ai-machine-learning/t...