3 ms·

How many tokens per second?

• mark_l_watson 2y ago
  I am away from my computer, but I think it was about 10/second - not too bad.
• andrewmunsell 2y ago
  Another data point: 17.6 tokens/s on an M4 Max 40 core GPU
• behnamoh 2y ago
  8.4 tps on M1 Pro chip with 32GM RAM (Q4 model, 18GB).