3 ms·
> And 5 tok/s with 65B Which is still incredible. I recently tested llama.cpp on a dual xeon server with 24 cores and 70gb RAM and on the 65B model it took lik
by geek_at 3y ago
> And 5 tok/s with 65B
Which is still incredible. I recently tested llama.cpp on a dual xeon server with 24 cores and 70gb RAM and on the 65B model it took like 15 seconds for one token.
GPU seems the way to go here
- samwillis 3y agoExactly, 5 tok/s is just fast enough for real-time speech.