3 ms·
But is it stable? I've seen a video on YouTube [0] where someone was showing LLM inferencing on a MacBook compared to an NVIDIA card, and it was really good, bu
by qwertox 3y ago
But is it stable? I've seen a video on YouTube [0] where someone was showing LLM inferencing on a MacBook compared to an NVIDIA card, and it was really good, but before the video was over, everything on the Mac stopped working (he was also recording via OBS I think and it went to a halt).
This was a MacBook, but since you're just saying "Mac", I don't know what you're referring to.
I think that the MacBook couldn't cope with the heat generated so it went to its knees. I've read about it on Reddit as well.
[0] https://youtu.be/jaM02mb6JFM?t=411 https://youtu.be/jaM02mb6JFM?t=411
- mark_l_watson 3y agoI bought a Mac Mini specifically for experimenting with LLMs. On the rare times I want to try fine tuning, I use Colab Pro or rent a server on Lambda Labs. I just use the M2 for inferencing, so far. So, no heat problems. EDIT: maybe the YouTube demo was using 2 bit quantization?
- bombcar 3y agoI've heard multiple reports that you can hard-lock a M1/2 Mac simply by using OpenGL in harsh ways, so I wouldn't be surprised that an LLM + OBS can crash it. I suspect OBS can crash it all by itself.
- ucm_edge 3y agoI set my kids up with Faraday.dev on their M2 Airs and it does just fine. They will have chatbots up for hours to help them with schoolwork or when playing Minecraft to give them ideas. The big thing is just don’t lock the models into memory on lower RAM systems. That gets you into trouble when some of the unified RAM is needed for something else and you can soft lock the system. The heat is not an issue though in my experience.
- teaearlgraycold 3y agoI use my MBP for local ML and then CUDA in production. So far so good. We run YOLO inference models and Open CLIP models. Soon adding SD1.5 to the mix!