4 ms·
They keep mentioning the RTX 3090 (with 24 GB VRAM), but the model is only 14.1 GB. Shouldn’t it fit a 5060 Ti 16GB, for instance?
by wtcactus 2y ago
They keep mentioning the RTX 3090 (with 24 GB VRAM), but the model is only 14.1 GB.
Shouldn’t it fit a 5060 Ti 16GB, for instance?
- oktoberpaard 2y agoWith a 128K context length and 8 bit KV cache, the 27b model occupies 22 GiB on my system. With a smaller context length you should be able to fit it on a 16 GiB GPU.
- jsnell 2y agoMemory is needed for more than just the parameters, e.g. the KV cache.
- cubefox 2y agoKV = key-value
- Havoc 2y agoJust checked - 19 gigs with 8k context @ q8 kv.Plus another 2.5-ish or so for OS etc. ...so yeah 3090