3 ms·
I assumed I needed a M1 pro or better due to RAM requirements, but I was able to run the 7B model on a 16GB M1 Mac Mini. The system reported 4GB RAM usage for t
by hnfong 4y ago
I assumed I needed a M1 pro or better due to RAM requirements, but I was able to run the 7B model on a 16GB M1 Mac Mini. The system reported 4GB RAM usage for the inference process. I suspect it's possible to run the 13B model as well.
- mrtksn 4y agoYou can, I was able to run 13B on my 16GB 8c8g M1 Air. The performance was 2-3 tokens/second. It felt on par with ChatGPT on a busy day.