4 ms·

For those of us with Twitter.com blocked in local dns, what size model is it getting 40tok/s on?

• samwillis 3y ago
  From the thread: > Watching llama.cpp do 40 tok/s inference of the 7B model on my M2 Max, with 0% CPU usage, and using all 38 GPU cores. > Getting 24 tok/s with the 13B model > And 5 tok/s with 65B
  • geek_at 3y ago
    > And 5 tok/s with 65B Which is still incredible. I recently tested llama.cpp on a dual xeon server with 24 cores and 70gb RAM and on the 65B model it took like 15 seconds for one token. GPU seems the way to go here
    • samwillis 3y ago
      Exactly, 5 tok/s is just fast enough for real-time speech.
• jazzyjackson 3y ago
  7B here's the linked github fwiw https://github.com/ggerganov/llama.cpp/pull/1642 https://github.com/ggerganov/llama.cpp/pull/1642
  • sitkack 3y ago
    The tweet is literally just this.
  • kristianp 3y ago
    That's a better link than the twitter posts.