8 ms·
Is this single-thread? Or are they putting all available CPUs on the problem?
by Animats 3y ago
Is this single-thread? Or are they putting all available CPUs on the problem?
- sharms 3y agoThe problem is memory bandwidth rather than CPU cores: "Memory bandwidth is the limiting factor in almost everything to do with sampling from transformers. Anything that reduces the memory requirements for these models makes them much easier to serve"
- brucethemoose2 3y agoIts complicated. If you dont have a GPU, prompt ingestion is totally threaded. The more cores the better. For generating tokens, more cores helps to a point, but then: - You start saturating the memory bus, and performance plateaus. - There is some overhead from the threading implementation, and too many threads hurts performance. The ideal number of threads varys per CPU. For instance, using hyperthreaded cores or Apple/Intel e cores typically hurts performance... But not always. You just have to test and see.
- regularfry 3y agoIt's multithreaded.