9 ms·
llama.cpp can run the Q4 variant of the same at 30tok/s on an M1 Pro unlike the 20tok/s being quoted
by iAkashPaul 3y ago
llama.cpp can run the Q4 variant of the same at 30tok/s on an M1 Pro unlike the 20tok/s being quoted
- rgbrgb 3y agoI get similar speeds on my m1, though it depends a bit on the quantization. If anyone else is on Mac and just want to have a play, FreeChat (my app) is a simple chat wrapper for llama.cpp. Clicking the little … on a message and checking the info will tell you the generation speed. [0]: https://www.freechat.run/ https://www.freechat.run/