4 ms·
Yesterday I tried mixtral 7bx8 running on the CPU. With an Intel 11th gen chip and 64gb DDR4 at 3200mhz, I got around 2-4 tokens/second in a small context, this
by tarruda 3y ago
Yesterday I tried mixtral 7bx8 running on the CPU. With an Intel 11th gen chip and 64gb DDR4 at 3200mhz, I got around 2-4 tokens/second in a small context, this gets progressively slower as the context grows.
You would get a much better experience with apple silicon and lots of RAM
- yobanate 3y agoCan confirm. My M3 Max gets about 22t/s, putting the bottleneck BKAC.
- 3abiton 3y agoThat's 10x speed increase. What's the secret behind apple M3? Faster clocked RAMs? Specific AI hardware?
- bugglebeetle 3y agoUnified memory and optimizations in llama.cpp (which Ollama wraps).
- ithkuil 3y agoIs that using the GPU?
- bugglebeetle 3y agoIt can be variably configured. There are details in the repo, but llama.cpp makes use of Metal.