4 ms·
A few months ago I bought a Mac with a M2 Pro 32G, and it is great for LLMs. I never thought that I would be running models like mixtral-8-7B-q3 locally on my o
by mark_l_watson 3y ago
A few months ago I bought a Mac with a M2 Pro 32G, and it is great for LLMs. I never thought that I would be running models like mixtral-8-7B-q3 locally on my own computer. I think the M2 Pro has 200GB/sec internal bandwidth, about 25% faster than the M3.
I think smaller models will keep getting better. For privacy and economic reasons, it makes sense to do as much AI on-device as possible.
- dpacmittal 3y agoI've had a similar experience with lm studio and draw things on my Macbook.
- pc86 3y agoCan you recommend any primers about running small models on-device, training/fine tuning on particular datasets (e.g. a specific code base), etc?
- erichocean 3y agohttps://github.com/ml-explore/mlx https://github.com/ml-explore/mlx https://github.com/ml-explore/mlx-examples https://github.com/ml-explore/mlx-examples This is where all of the action is at right now. You can also use LM Studio: https://lmstudio.ai/ https://lmstudio.ai/
- ParetoOptimal 3y agoAn open source lmstudio software released recently, but I'm having trouble finding a link.
- pmx 3y agoyou might be thinking of GPT4ALL? https://gpt4all.io/index.html https://gpt4all.io/index.html I struggled to get it working on my machine, it kept corrupting the models as it downloaded them.
- dchuk 3y agoWas it this? I haven’t tried it yet but it does look nice. https://jan.ai/ https://jan.ai/
- ParetoOptimal 3y agoYes! That was it.
- rgbrgb 3y agoI maintain this one for native macOS chat https://github.com/psugihara/FreeChat https://github.com/psugihara/FreeChat
- mark_l_watson 3y agoI tried everything for running small models locally, and finally settled on ollama.ai - good command line and REST API functionality. If you like Lisp, recent updates to my Racket and Common Lisp books have examples using the Ollama REST APIs (Can be read online).
- qwertox 3y agoBut is it stable? I've seen a video on YouTube [0] where someone was showing LLM inferencing on a MacBook compared to an NVIDIA card, and it was really good, but before the video was over, everything on the Mac stopped working (he was also recording via OBS I think and it went to a halt). This was a MacBook, but since you're just saying "Mac", I don't know what you're referring to. I think that the MacBook couldn't cope with the heat generated so it went to its knees. I've read about it on Reddit as well. [0] https://youtu.be/jaM02mb6JFM?t=411 https://youtu.be/jaM02mb6JFM?t=411
- mark_l_watson 3y agoI bought a Mac Mini specifically for experimenting with LLMs. On the rare times I want to try fine tuning, I use Colab Pro or rent a server on Lambda Labs. I just use the M2 for inferencing, so far. So, no heat problems. EDIT: maybe the YouTube demo was using 2 bit quantization?
- bombcar 3y agoI've heard multiple reports that you can hard-lock a M1/2 Mac simply by using OpenGL in harsh ways, so I wouldn't be surprised that an LLM + OBS can crash it. I suspect OBS can crash it all by itself.
- ucm_edge 3y agoI set my kids up with Faraday.dev on their M2 Airs and it does just fine. They will have chatbots up for hours to help them with schoolwork or when playing Minecraft to give them ideas. The big thing is just don’t lock the models into memory on lower RAM systems. That gets you into trouble when some of the unified RAM is needed for something else and you can soft lock the system. The heat is not an issue though in my experience.
- teaearlgraycold 3y agoI use my MBP for local ML and then CUDA in production. So far so good. We run YOLO inference models and Open CLIP models. Soon adding SD1.5 to the mix!