2 ms·

6.3 tok/s has been demonstrated on q4_0 Falcon 180B on the 192gb Mac studio: https://x.com/ggerganov/status/1699791226780975439?s=46&t=rumiybW6rwwWri-_imGg0Q ht

• staticman2 3y ago
  A lot of people are reporting low context sizes tokens per second without discussing how slow it is at bigger sizes. In some cases they are also not mentioning time to first inference. If you dig around /LocalLLaMA you can find some posts with better bench-marking.