3 ms·
Anyone know if this (or any LLM) is worth running locally on a Macbook M2 Air? I'll give it a shot over the weekend but if anyone knows I'm curious!
by youniverse 3y ago
Anyone know if this (or any LLM) is worth running locally on a Macbook M2 Air?
I'll give it a shot over the weekend but if anyone knows I'm curious!
- buffington 3y agoWithout knowing what you consider to be of worth, that's difficult to answer. The good news is that the article describes in detail how to determine for yourself if it's worth it. The author uses an M2 as well, so you can at least know it'll likely work.
- MPSimmons 3y agoIt depends on what you want them to do and what you want to get out of them. A lot of people use less machine than a M2 Air for different things.
- brandall10 3y agoIf you have 8GB you can play around with heavily quantized 7B models, or up to moderately quantized 13GB models w/ 16GB. Something like a Mistral-7B Dolphin finetune actually is surprisingly useful, like GPT-3.5 in some respects. I imagine it would render ~10 tok/sec on an M2 for at least short bursts until throttling sets in. As I mentioned in another post, try this out with LM Studio. Super simple GUI that even a non-tech person could probably figure out for finding/downloading/loading models w/ a ChatGPT like interface for chatting. https://lmstudio.ai/ https://lmstudio.ai/
- iJohnDoe 3y agoIs there any reason to run this if you already have LM Studio installed? Which LLM to download for general questions/info that has good accuracy? Separately, I couldn’t figure out how to use the NVIDIA GPU with LM Studio. Only wants to use the Intel card. I use NVIDIA with Diffusion with no problems if that makes any difference.
- dataveg 3y agoI have Ollama running llama 7B on an 32GB Mac M1 Pro and its fast enough to be useful.