3 ms·
LLaMA-13B 4bit is around 7GB and beats GPT-3.5 175B in benchmarks. It runs at acceptable speeds on a Macbook Air M1 or a $100 consumer video card.
by MacsHeadroom 4y ago
LLaMA-13B 4bit is around 7GB and beats GPT-3.5 175B in benchmarks.
It runs at acceptable speeds on a Macbook Air M1 or a $100 consumer video card.
- diimdeep 4y agoThere is a drop in performance after quantization, so initial benchmarks aren't relevant. Here are some benchmarks for quantized https://github.com/qwopqwop200/GPTQ-for-LLaMa https://github.com/qwopqwop200/GPTQ-for-LLaMa