3 ms·

Multithreading support in llama.cpp is probably still pretty busted, assuming it uses the same underlying NN inference code as whisper.cpp: https://github.com/g

• imtringued 3y ago
  From what I have heard they use manual spin locks. Generally, spin locks are not a good idea unless you want to dedicate the entire machine to a single application. If the process a spinlock waits on gets suspended, you're burning CPU time for nothing. The OS thinks a spinlock making zero progress is actually a high priority process, so it is starving the suspended process from making progress.
  • Ono-Sendai 3y ago
    Yeah the code looks like a spinlock. It behaves terribly under contention, resulting in performance falling off a cliff as the number of threads increases. Adding more threads actually slows down the total performance. I would fix it if I could be bothered. Instead I will just use the Cuda whisper backend which is pretty nice and fast.