4 ms·
Can confirm. My M3 Max gets about 22t/s, putting the bottleneck BKAC.
by yobanate 3y ago
Can confirm.
My M3 Max gets about 22t/s, putting the bottleneck BKAC.
- 3abiton 3y agoThat's 10x speed increase. What's the secret behind apple M3? Faster clocked RAMs? Specific AI hardware?
- bugglebeetle 3y agoUnified memory and optimizations in llama.cpp (which Ollama wraps).
- ithkuil 3y agoIs that using the GPU?
- bugglebeetle 3y agoIt can be variably configured. There are details in the repo, but llama.cpp makes use of Metal.