3 ms·
I told you a while ago that llama.cpp supports 4/8-bit (and other) quantization just fine (and fast) on x86_64.
by gliptic 3y ago
I told you a while ago that llama.cpp supports 4/8-bit (and other) quantization just fine (and fast) on x86_64.
- Roark66 3y agoYes, you did. Back then I went to their github page and I found no mention of x86. Now I see it is there. Great. I'll give it a try. Also I see something much better. The same guy behind llama.cpp authors a universal ml library that does cpu quantization. I wasn't aware of that and it is far more impressive than running a couple of chosen models.