20 ms·
Well the Llama.cpp running on CPUs with decent speed and fast development improvements, hints towards CPUs. And there the size of the model is less important as
by tommek4077 3y ago
Well the Llama.cpp running on CPUs with decent speed and fast development improvements, hints towards CPUs. And there the size of the model is less important as the RAM is the limit. At least for interference this is now a viable alternative.
- redox99 3y agoOutside of Macs, llama.cpp running fully on the cpu is more than 10x slower than a GPU.
- tommek4077 3y agoBut having 32 real cores in a cpu is so much cheaper than having mumtiple gpus. RAM is also much cheaper as VRAM.
- kirill5pol 3y agoFor local yes, but at data center level the parallelization is still often worth it.