3 ms·
Sorry, but the topic of this post, llama.cpp, runs quantized 4/8 bit models just fine on x86_64 with AVX2, or am I missing some requirement you have?
by gliptic 3y ago
Sorry, but the topic of this post, llama.cpp, runs quantized 4/8 bit models just fine on x86_64 with AVX2, or am I missing some requirement you have?
- Roark66 3y agoWait, I wasn't aware llama.cpp even runs on x86_64.i thought it is arm hw only. If what you say is correct that indeed is very interesting. Especially if I can extend it to other models like falcon.
- gliptic 3y agoIt doesn't support Falcon right now, but there's a fork that does (https://github.com/cmp-nct/ggllm.cpp/ https://github.com/cmp-nct/ggllm.cpp/).