3 ms·
I’ve heard rumbling about native context length. I don’t know too much about it, but is this natively 1M context length? So even models like llama3 8b say they
by bloomingkales 2y ago
I’ve heard rumbling about native context length. I don’t know too much about it, but is this natively 1M context length?
So even models like llama3 8b say they have a larger context, but they really don’t in practice. I have a hard time getting past 8k on 16gb vram (you can definitely set the context length higher, but the quality and speed degradation is obvious).
I’m curious how people are doing this on modest hardware.
- segmondy 2y agoYou can't on modest hardware, VRAM size is a function of model size, KV cache that depends on context length and the quant size of the model and K/V. 16gb isn't much really. You need more vram, the best way for most folks is to buy a macbook with unified memory. You can get a 128gb mac, but it's not cheap. If you are handy and resourceful you can build a GPU cluster.
- A4ET8a8uTh0_v2 2y agoI never thought I would say it, but the 128gb mbp is probably the most cost efficient way ( and probably easiest ) of doing it. New nvidia cards ( 5090 ) are 32gb and supposedly just shy of 2k and used a100 40gb is about 8k.. All in all, not a cheap hobby ( if you are not doing it for work ).
- elorant 2y agoYou need a model that has specifically been extended for larger context windows. For Llama-3 there's Llama3-gradient with up to 1M tokens. You can find it at ollama.com