3 ms·
this can apparently run on 48GB
by seydor 3y ago
this can apparently run on 48GB
- lolinder 3y agoWhen quantized to 4 bits, yes. You lose some quality by doing that, though, as compared to the full f16.
- evilduck 3y agoFrom what I've gathered when reading up on this topic, if RAM is your constraint the common thought has been that higher parameter models quantized down to smaller sizes will outperform lower parameter models running at higher quantization, i.e. it may still be preferable to use the 70B Llama model quantized to 4-bits than something like an unquantized f16 Falcon 40B or the "coming soon" f16 33B Llama2.
- lolinder 3y agoYes, that is true! But you lose enough performance that comparisons to GPT-3.5 stop working.
- treprinum 3y ago2xA6000 NVLinked Ampere can run 70B 8-bit which is almost as good as fp16. I bought another A6000 just for that.