4 ms·
Huh, I've fought with a few of these things on my laptop, no nvidia GPU, limited ram, etc. This actually worked as advertised.
by traverseda 3y ago
Huh, I've fought with a few of these things on my laptop, no nvidia GPU, limited ram, etc.
This actually worked as advertised.
- 3abiton 3y agoWhat's the performance like (quality and speed wise)?
- all2 3y agoMistral 7b is serviceable for short contexts. If you have a longer conversation, token generation can start to lag a lot.
- tarruda 3y agoYesterday I tried mixtral 7bx8 running on the CPU. With an Intel 11th gen chip and 64gb DDR4 at 3200mhz, I got around 2-4 tokens/second in a small context, this gets progressively slower as the context grows. You would get a much better experience with apple silicon and lots of RAM
- yobanate 3y agoCan confirm. My M3 Max gets about 22t/s, putting the bottleneck BKAC.
- 3abiton 3y agoThat's 10x speed increase. What's the secret behind apple M3? Faster clocked RAMs? Specific AI hardware?
- bugglebeetle 3y agoUnified memory and optimizations in llama.cpp (which Ollama wraps).
- ithkuil 3y agoIs that using the GPU?
- bugglebeetle 3y agoIt can be variably configured. There are details in the repo, but llama.cpp makes use of Metal.