3 ms·
Not really, no. You want to use the GPU not the CPU. Macs are neat here since they can use the shared memory with a rather high bandwidth. So even if the GPU is
by diffeomorphism 2y ago
Not really, no. You want to use the GPU not the CPU. Macs are neat here since they can use the shared memory with a rather high bandwidth. So even if the GPU is much slower, the ram is much worse than proper vram, and ridiculously overpriced for that,... often the bottlenecks are ram amount and bandwidth.
- yjftsjthsd-h 2y agoHaving run ollama on CPU: Yes, it's just slower. Not even intolerably slow IMO, though I used small models and don't mind some turnaround time.
- jart 2y agoI've seen llamafile go about 10x faster on CPU if you try it.
- yjftsjthsd-h 2y agoReally! Same model and everything? I guess I need to go benchmark them - 10x would legit obviate GPU for me
- deleted 2y ago[deleted]