Its complicated. If you dont have a GPU, prompt ingestion is totally threaded. The more cores the better. For generating tokens, more cores helps to a point,

Its complicated. If you dont have a GPU, prompt ingestion is totally threaded. The more cores the better. For generating tokens, more cores helps to a point, but then: - You start saturating the memory bus, and performance plateaus. - There is some overhead from the threading implementation, and too many threads hurts performance. The ideal number of threads varys per CPU. For instance, using hyperthreaded cores or Apple/Intel e cores typically hurts performance... But not always. You just have to test and see.