2 ms·
Yes (I just don't have that much ram) I have a separate branch that streams weights from ram - at which point I think I was only seeing negligible performance
by gpm 4y ago
Yes (I just don't have that much ram)
I have a separate branch that streams weights from ram - at which point I think I was only seeing negligible performance loss compared to storing the weights in vram. The bottleneck was compute, not GPU bandwidth.
- MacsHeadroom 4y agoThe 65B model only needs just over 32GB of VRAM to run. It does not need system RAM to run/use if you use pre-quantized weights which you can find many places already. No need to quantize yourself (besides it takes almost a day to do 4bit GPTQ quantization on 3xA6000).
- gpm 4y agoQuantizing is a lossy process, you can't really claim to be running the 65B model llama at that point (though the 65b qgpt-llama does look like it might be very useful)
- Tepix 4y agoThe GPTQ paper https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323 claims "negligible accuracy degradation relative to the uncompressed baseline".
- deleted 4y ago[deleted]
- Tepix 4y agoAre you sure? I think it took a mere 2 hours to do 4bit GPTQ quantization of LLaMA-65B on 1x RTX 3090. But i may be mistaken.