4 ms·
The 65B model only needs just over 32GB of VRAM to run. It does not need system RAM to run/use if you use pre-quantized weights which you can find many places a
by MacsHeadroom 4y ago
The 65B model only needs just over 32GB of VRAM to run. It does not need system RAM to run/use if you use pre-quantized weights which you can find many places already.
No need to quantize yourself (besides it takes almost a day to do 4bit GPTQ quantization on 3xA6000).
- gpm 4y agoQuantizing is a lossy process, you can't really claim to be running the 65B model llama at that point (though the 65b qgpt-llama does look like it might be very useful)
- Tepix 4y agoThe GPTQ paper https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323 claims "negligible accuracy degradation relative to the uncompressed baseline".
- deleted 4y ago[deleted]
- Tepix 4y agoAre you sure? I think it took a mere 2 hours to do 4bit GPTQ quantization of LLaMA-65B on 1x RTX 3090. But i may be mistaken.