4 ms·
Compared to a GPU like a 4090 with equal vram it probably won't fare well as others point out, however it far outperforms any CPU. On an M1 Ultra MacBook Pro I
by gkfasdfasdf 2y ago
Compared to a GPU like a 4090 with equal vram it probably won't fare well as others point out, however it far outperforms any CPU. On an M1 Ultra MacBook Pro I was seeing like 40 tokens/second with llama3:7B vs 9 tokens/second on various Intel servers/desktops with sufficient ram.