4 ms·
I think using this project https://github.com/ggerganov/llama.cppav https://github.com/ggerganov/llama.cppav on a CPU machine with AVX instructions would be a
by jerrygenser 3y ago
I think using this project https://github.com/ggerganov/llama.cppav https://github.com/ggerganov/llama.cppav
on a CPU machine with AVX instructions would be a better bang for your buck than GPU. Depends on if your use case can tolerate the latency