2 ms·
That's very exciting. I'm building it right now. Can anyone explain how llama.cpp works? On a very high level, I'm interested in how it's possible to run it on
by heap_perms 4y ago
That's very exciting. I'm building it right now.
Can anyone explain how llama.cpp works? On a very high level, I'm interested in how it's possible to run it on CPU.
- woodson 4y agoIt quantizes 32-bit floating point weights to 4-bit integers so that models fit into memory and run reasonably fast on a CPU by using the processor’s vector extensions.