5 ms·
Running mixtral on an 32gb M3 Max is surprisingly hard on LM Studio... turning on "Apple Metal" mode just crashes it. It takes about 10 minutes to load otherwis
by yawnxyz 3y ago
Running mixtral on an 32gb M3 Max is surprisingly hard on LM Studio... turning on "Apple Metal" mode just crashes it. It takes about 10 minutes to load otherwise into memory, and inference is super slow.
Anyone got suggestions? Would love for it to run summarization over GBs of ncbi data.
- smcleod 3y ago32GB memory only leaves about 24-26GB for the GPU by default which is quite low for a larger model like that. For comparison it runs great on a M2 Max 96GB.
- minimaxir 3y agoDo you need to use Mixtral? The typical quantized Mistral 7B and similar models can do summarization fine.
- ithkuil 3y agoI'm using gpt-4 as an assistant for basic computing questions such helping with quick shell one liners or looking up for the simplest way to do some manipulation of rust values (vec of result into result of vec which I always forget how to do etc). I wanted to try how local models compare with that use case of mine. The only one that worked decently was Mixtral
- yawnxyz 3y agoHaha no not at all, but it’s fun!
- sunpazed 3y agoTry a quantized model on llama.cpp — I have an M1 Max 32Gb and it’s running fairly fast in 20Gb of memory.
- mark_l_watson 3y agoIt runs OK on my 32B Mac Mini, but I have mostly been running it using the inexpensive Mistral and AllScale APIs.
- SillyUsername 3y agoGet a dirt cheap windows machine with >32GB or rent a VM?