3 ms·
Won't the 65b model (almost) fit into 128GB RAM? Or into 128GB RAM and 24GB VRAM?
by Tepix 4y ago
Won't the 65b model (almost) fit into 128GB RAM?
Or into 128GB RAM and 24GB VRAM?
- MacsHeadroom 4y agoLLaMA-65B fits in 32GB of VRAM using state of the art GPTQ quantization with no output performance loss. https://github.com/qwopqwop200/GPTQ-for-LLaMa https://github.com/qwopqwop200/GPTQ-for-LLaMa
- minxomat 4y agoSo if I'm reading this right, 65B at 4bit would consume around 20GB of VRAM and ~130GB of system RAM?
- MacsHeadroom 4y agoLLaMA it doesn't require any system RAM to run. It requires some very minimal system RAM to load the model into VRAM and to compile the 4bit quantized weights. But if you use pre-quantized weights (get them from HuggingFace or a friend) then all you really need is ~32GB of VRAM and maybe around 2GB of system RAM for 65B. (It's 30B which needs 20GB of VRAM.)
- minxomat 4y agoThe full use case includes quantisation, which the repo points out uses a large amount of system RAM. Of course that’s not required if you skip that step.
- MacsHeadroom 4y agoJudging from downloads of the 4bit file and how many people I've seen post about quantizing it themselves, around 99% of people are just downloading the pre-quantized files. I would not personally call compilation of software part of its "use case." It's use case is text generation.
- nl 4y agoQuantisation is a once off process. I suspect most people who don't have access to a machine with enough RAM and don't want to use the pre-quantized version can afford the $20 to hire a big cloud server for an day. Or it is probably possible to make it work slowly using a swapfile on Linux.
- Tepix 4y agoCloser to 38-40GB VRAM (and hardly any RAM).
- gpm 4y agoYes (I just don't have that much ram) I have a separate branch that streams weights from ram - at which point I think I was only seeing negligible performance loss compared to storing the weights in vram. The bottleneck was compute, not GPU bandwidth.
- MacsHeadroom 4y agoThe 65B model only needs just over 32GB of VRAM to run. It does not need system RAM to run/use if you use pre-quantized weights which you can find many places already. No need to quantize yourself (besides it takes almost a day to do 4bit GPTQ quantization on 3xA6000).
- gpm 4y agoQuantizing is a lossy process, you can't really claim to be running the 65B model llama at that point (though the 65b qgpt-llama does look like it might be very useful)
- Tepix 4y agoThe GPTQ paper https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323 claims "negligible accuracy degradation relative to the uncompressed baseline".
- deleted 4y ago[deleted]
- Tepix 4y agoAre you sure? I think it took a mere 2 hours to do 4bit GPTQ quantization of LLaMA-65B on 1x RTX 3090. But i may be mistaken.