2 ms·
From the thread: > Watching llama.cpp do 40 tok/s inference of the 7B model on my M2 Max, with 0% CPU usage, and using all 38 GPU cores. > Getting 24 tok/s wi
by samwillis 3y ago
From the thread:
> Watching llama.cpp do 40 tok/s inference of the 7B model on my M2 Max, with 0% CPU usage, and using all 38 GPU cores.
> Getting 24 tok/s with the 13B model
> And 5 tok/s with 65B
- geek_at 3y ago> And 5 tok/s with 65B Which is still incredible. I recently tested llama.cpp on a dual xeon server with 24 cores and 70gb RAM and on the 65B model it took like 15 seconds for one token. GPU seems the way to go here
- samwillis 3y agoExactly, 5 tok/s is just fast enough for real-time speech.