4 ms·
It might run the smaller flash version, but 96gb is not enough for the trillion-parameter model. The M3 Ultra's GPU is a bit on the weak side for large-scale i
by bigyabai 7mo ago
It might run the smaller flash version, but 96gb is not enough for the trillion-parameter model.
The M3 Ultra's GPU is a bit on the weak side for large-scale inference, so you'll be waiting on token prefill for most coding/agent workflows.
- namegulf 7mo agoThey have a 512gb ram option but pricey. Have you tried any other models with this M3 Ultra?
- bigyabai 7mo agoThe 512gb model would have to use a lobotomized quant like q_2 or q_1, and you would still be waiting 3-5 minutes to process context lengths in the 32,000-64,000 token range. Apple's GPUs are just not very fast for inference. I'd stick to the smaller 7b-18b parameter range or MOE models like Qwen if you want a usable inference speed.