3 ms·
128GB on a Ryzen 395 is enough but not by a lot. Around 90GB for weights plus 10GB to 20GB for KV cache and checkpoints. I don't know about 64GB of VRAM for les
by hedgehog 2mo ago
128GB on a Ryzen 395 is enough but not by a lot. Around 90GB for weights plus 10GB to 20GB for KV cache and checkpoints. I don't know about 64GB of VRAM for less than around $2500 by itself.
- bitexploder 2mo agoTesla V100? Would have to be a 3-ish bit quant in 64GB ram