4 ms·
Yes, RAM requirement is BnL same for GPU and using the metal/GPU in Apple Silicon. Running LLM models on a MacBook Pro with Apple Silicon vs. a PC with an Nvid
by Terretta 2y ago
Yes, RAM requirement is BnL same for GPU and using the metal/GPU in Apple Silicon.
Running LLM models on a MacBook Pro with Apple Silicon vs. a PC with an Nvidia 4090 GPU has trade-offs. My 128GB MacBook Pro handles models using up to 96GB of unified memory, running at a little under half the speed of a 4090. If you use a quantized version of full floating point model, you can run the largest open models available.
While the 4090 has 24GB of dedicated memory and higher bandwidth (1000 GB/s vs. 400 GB/s on M3 Max), the Mac’s unified memory system (up to 128GB) is flexible and holds smarter models (8 bit and 6 bit models act still mostly all there, 4 bit is so so, 2 bit is brain damaged).
The M2 Ultra in Mac Studio offers even more (800 GB/s bandwidth and 192GB memory). So, ok, 6 or 8 of 4090 cards or 4 x A6000 cards excels in raw performance, but Apple’s unified memory in a laptop fits in your backback.
It's not clear to me why Macbooks and Mac Studio Ultras with maxed out RAM aren't selling better if you look at the convenience and price relative to model size. Models that fit in one 4090 or even a pair of 4090s are toys compared to what fits on these, so for the big models you're comparing a laptop to a minifridge.
- tyfon 2y agoI have a 5940x with 128 gb ram. It's a bit slower perhaps than the mac, but i get the best of both worlds. That is I get a lot of RAM to hold the model and I can offload as much of it as possible to the GPU. This works especially well with models like mixtral 8x22, but also models like llama3 and the old large bloom model. I also get the utility of running Linux instead of the closed up mac os. But running large models locally is not exclusive to mac studio, you can do the same on PC for a much lower cost.
- Terretta 2y agoI get the utility of a laptop that runs 20 hours on battery and slips in the side pocket of my carry-on or shoulder bag. (The Mac can also split between RAM and GPU.) Mixtral 8x22 and Llama 3 70b stream at roughly the same speed as last year's GPT-4. > closed up MacOS https://github.com/apple-oss-distributions/distribution-macOS https://github.com/apple-oss-distributions/distribution-macO... curl https://alx.sh | sh https://asahilinux.org/ https://asahilinux.org/ I prefer the "utility" of BSDs, but that's just a preference.
- talldayo 2y agoAsahi is Fischer-Price tiers of support compared to what even Nvidia, the most loathed OEM on Linux, provides for free to their users. If that's the best option available, it should be no wonder that server customers are avoiding Apple like the plague. Apple has to beg their audience to reverse-engineer their own OpenCL drivers if they want them; Nvidia ships them alongside CUDA. These two companies are not the same. Have you ever seen the inside of a datacenter? Why is it that surprising to you that nobody perks up when you start waxing on about battery life? Even terms of power-to-performance, Apple's latest chips get ethered by Nvidia's server offerings. This "Apple for Inference" meme is so dead that I can only feel sad when I see people unironically promoting it. You actually think serious customers are going to load up Asahi (even funnier, MacOS) on their Mac Pro... so they can inference half as fast as a single Blackwell GPU? You think the industry is doing this shit? I don't even think the Steve Jobs apologists are dumb enough to fall for this one, you must be a particularly aspirational shareholder.
- daghamm 2y ago"It's not clear to me why Macbooks and Mac Studio Ultras with maxed out RAM aren't selling" Aren't these machines extremly expensive and generally not upgradable?
- Terretta 2y agoThey cost less than the 4 - 8 graphics cards and monster desktop PC that would be needed, plus, hey, it's a laptop.
- deleted 2y ago[deleted]