8 ms·
Llama 7B is infinite times smarter than current Siri though. And with 3 bit quantization it is like 3GB of RAM which is completely achievable.
by YetAnotherNick 3y ago
Llama 7B is infinite times smarter than current Siri though. And with 3 bit quantization it is like 3GB of RAM which is completely achievable.
- _c8fm 3y agoHow much does 3 bit quantization affect performance?
- YetAnotherNick 3y agoAccording to few papers and https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684, ~2.5GB 7B parameters model size has the same performance as baseline 7B model(with 14GB size).