4 ms·
The execution speed for LLM inference gets so slow once you reach models that even fill out a 64GB machine. I was tempted, too, but then realized it was unusab
by novok 2y ago
The execution speed for LLM inference gets so slow once you reach models that even fill out a 64GB machine. I was tempted, too, but then realized it was unusable past 48GB-ish and stacking used 3090s was the best price / perf / vram ratio.
What are the good use cases for very large memory amounts?
- behnamoh 2y agoI ran all my research papers (about LLMs) on a Mac Studio.
- reitzensteinm 2y agoMixture of Expert models, where all parameters must be in memory but only a subset are accessed per token, are a sweet spot for Macs. DeepSeek v3 for instance has 671B params, but should have the memory bandwidth of a 37B dense model with a batch size of one.
- rpastuszak 2y agoAny model you'd recommend for a Mac with 48 GB RAM?
- reitzensteinm 2y ago48GB is maybe just enough to squeeze a quantized 70B model in, like Llama 3.3, but you'll need to raise the GPU memory allocation limit [1] and it might not be super fast. You could also try Qwen 2.5 32b, which you should just work with ollama or LM Studio with no config changes. I've got a 32gb M1 Max and a 24gb 4090, and I barely ever run models on my Mac, as the memory bandwidth and compute for prefill is much better on the 4090. But I'm essentially locked out of Llama 3 70B class models, which I only use via API. [1] See: https://www.reddit.com/r/LocalLLaMA/comments/186phti/m1m2m3_increase_vram_allocation_with_sudo_sysctl/ https://www.reddit.com/r/LocalLLaMA/comments/186phti/m1m2m3_...
- Terretta 2y agoUnusable in what way? Llama 3.3 70b q8 with 100k+ context runs as well as GPT 3.5 did a couple years back, except local on Mac and smarter.
- lumost 2y agoLarge context has different perf than large model. Op was likely thinking of running 400b models and finding the compute wasn’t enough to make the memory useful.
- imtringued 2y agoIf you don't have tensor parallelism even 8 x A100 is going to struggle with that kind of model.
- novok 2y agoLook on youtube running various models and even a 70b model does a slow 3tok/s
- shikon7 2y agoIt's slower than using just GPU RAM, but it's still faster than using a regular PC that has a much more limited bandwith between the main memory and the GPU. It's some sort of middle ground on how fast and how cheap you can do inference with LLMs that don't fit into a consumer GPU's RAM.
- smcleod 2y agoIt's definitely usable past 48GB, I have a 96GB M2 Max and regularly run models that use around 70GB that are very usable. I also have a home server with 2x3090 and 2xA4000 (80GB vRAM) - yes it's a lot faster, but it's a pain in the ass to build, it takes up a lot of space, uses 10x the power, and honestly - cost about the same as my MacBook Pro.