4 ms·
The unified memory ought to be great for running LLaMA on the GPU on these Macbooks (since it can't run on the Neural Engine currently) The point of llama.cpp
by anentropic 4y ago
The unified memory ought to be great for running LLaMA on the GPU on these Macbooks (since it can't run on the Neural Engine currently)
The point of llama.cpp is most people don't have a GPU with enough RAM, Apple unified memory ought to solve that
Some people have it working apparently:
https://github.com/remixer-dec/llama-mps https://github.com/remixer-dec/llama-mps
- rnk 4y agoThank you, that's exactly what I was looking for, specific info on perf.
- anentropic 4y agoI think the GPU performance for inference is probably limited currently by immaturity of PyTorch MPS (Metal) backend before I found the repo above I had a naive attempt to get llama running with mps and it didn't "just work" - bunch of ops not supported etc