3 ms·
That is likely a bug. There is nothing in the maths that should make prompt tokens slower on CPU.
by londons_explore 4y ago
That is likely a bug. There is nothing in the maths that should make prompt tokens slower on CPU.
- loufe 4y agoIt is a bug, it's been clearly discussed under the issues of LLaMA.cpp. The wait time is far from atrocious, with a 5600x and the 13B I would wait 15 seconds before input starts for a relatively complex prompt.