3 ms·
I've been using Gemini 1.5 with 1M tokens and it's a totally different game. You can essentially "train" the model on whatever hyper specific stuff you have on
by Workaccount2 3y ago
I've been using Gemini 1.5 with 1M tokens and it's a totally different game. You can essentially "train" the model on whatever hyper specific stuff you have on hand.
Don't know how to work with my test system? No problem, here is an 800 page reference manual. Now you know.
- rahimnathwani 3y agoI'm curious what your setup is? I mean what platform, how much RAM and which particular version of the model?
- lennxa 3y agoThey are talking about Google's Gemini, not running locally.
- rahimnathwani 3y agoOops I confused Gemini and Gemma.
- theolivenbaum 3y agoWhat kind of latency do you observe when using Gemini with such large contexts?
- dmazin 3y agoInference takes 30+ seconds before it starts outputting even the first token.
- danielbln 3y agoHow's recall over 1M tokens?
- Workaccount2 3y agoAround 250k tokens its around 15-20 seconds. When you really pack it full (which I have only done just to test it) it can take a minute before it outputs anything. It also sometimes glitches out with massive contexts, and just stops generation mid sentence. Asking it to repeat the response clears it up though.
- phyalow 3y agohttps://youtu.be/6AOpomu9V6Q?si=ovW6Ok8vhxENW1JX https://youtu.be/6AOpomu9V6Q?si=ovW6Ok8vhxENW1JX
- darkteflon 3y agoCould you clarify whether you’re context stuffing the whole document on every query, or something else?
- sporkland 3y agoYeah looks like they misused the word "train". And instead meant context stuffing.
- ghoul2 3y agoI agree, definitely a game changer. Finally made me a convert. re latency: yes, latency can get quite high, for large contexts. minutes, 10s of minutes when you try to max out the context.