9 ms·
Apple M5 could ditch unified memory architecture for split CPU and GPU designs
- vimy 2y ago> That being said, such a split design may help with improved AI inferencing. Unified memory is the only reason Macs are so coveted right now for local AI. A single 192 gb ram Mac costs less than the equivalent in standalone GPUs.
- Tostino 2y agoI'm sorry, I think you mispronounced market segmentation... This is Apple we're talking about.
- novok 2y agoThe execution speed for LLM inference gets so slow once you reach models that even fill out a 64GB machine. I was tempted, too, but then realized it was unusable past 48GB-ish and stacking used 3090s was the best price / perf / vram ratio. What are the good use cases for very large memory amounts?
- behnamoh 2y agoI ran all my research papers (about LLMs) on a Mac Studio.
- reitzensteinm 2y agoMixture of Expert models, where all parameters must be in memory but only a subset are accessed per token, are a sweet spot for Macs. DeepSeek v3 for instance has 671B params, but should have the memory bandwidth of a 37B dense model with a batch size of one.
- rpastuszak 2y agoAny model you'd recommend for a Mac with 48 GB RAM?
- reitzensteinm 2y ago48GB is maybe just enough to squeeze a quantized 70B model in, like Llama 3.3, but you'll need to raise the GPU memory allocation limit [1] and it might not be super fast. You could also try Qwen 2.5 32b, which you should just work with ollama or LM Studio with no config changes. I've got a 32gb M1 Max and a 24gb 4090, and I barely ever run models on my Mac, as the memory bandwidth and compute for prefill is much better on the 4090. But I'm essentially locked out of Llama 3 70B class models, which I only use via API. [1] See: https://www.reddit.com/r/LocalLLaMA/comments/186phti/m1m2m3_increase_vram_allocation_with_sudo_sysctl/ https://www.reddit.com/r/LocalLLaMA/comments/186phti/m1m2m3_...
- Terretta 2y agoUnusable in what way? Llama 3.3 70b q8 with 100k+ context runs as well as GPT 3.5 did a couple years back, except local on Mac and smarter.
- lumost 2y agoLarge context has different perf than large model. Op was likely thinking of running 400b models and finding the compute wasn’t enough to make the memory useful.
- imtringued 2y agoIf you don't have tensor parallelism even 8 x A100 is going to struggle with that kind of model.
- novok 2y agoLook on youtube running various models and even a 70b model does a slow 3tok/s
- shikon7 2y agoIt's slower than using just GPU RAM, but it's still faster than using a regular PC that has a much more limited bandwith between the main memory and the GPU. It's some sort of middle ground on how fast and how cheap you can do inference with LLMs that don't fit into a consumer GPU's RAM.
- smcleod 2y agoIt's definitely usable past 48GB, I have a 96GB M2 Max and regularly run models that use around 70GB that are very usable. I also have a home server with 2x3090 and 2xA4000 (80GB vRAM) - yes it's a lot faster, but it's a pain in the ass to build, it takes up a lot of space, uses 10x the power, and honestly - cost about the same as my MacBook Pro.
- gazchop 2y agoI don’t think the principal market actually cares about that.
- mrtksn 2y agoWhen LLMs and diffusion models began spreading freely, it was very funny to have support for some beefy high end GPUs and MacBook Air M1. I hope Apple sticks with the architecture. Even if its not very practical, its great to have it as possible.
- halJordan 2y agoThis isn't splitting the unified memory, it's splitting the soc into a sip. The ram is still on the same interposer/substrate. The actual mm distance won't be regressing to say the distance of a dimm socket to the cpu socket. The software will still see a single memory pool I know im disagreeing with the article
- UncleOxidant 2y agoYeah, this is the edge Macs have right now in the AI space. It's why people are looking forward to the Strix Halo from AMD as it will also have a unified memory architecture and will probably cost a good bit less than a Mac.
- wtallis 2y ago> Another intriguing aspect is the separate CPU and GPU design. If true, this essentially means M5 will not use a unified memory architecture (UMA) shared between the CPU and the GPU. This does not follow. Intel is shipping unified memory processors with CPU cores and GPU cores on separate chiplets but still sharing the same memory controller (on a third chiplet, for Meteor Lake and Arrow Lake). AMD is about to launch Strix Halo, a high-end mobile processor that is rumored to consist of one or two CPU chiplets and an IO die with a big GPU and 256-bit memory controller.
- jsheard 2y agoPlus Apple themselves are already kind of doing it with the M Ultras - those are two chiplets glued together, each of which is only connected to half of the systems memory directly, but it still behaves like unified memory even though half of the memory traffic has to be routed via the other chiplet.
- wtallis 2y agoAnd in the datacenter space, AMD has taken things even further with the MI300A: > Twenty-four x86-architecture ‘Zen 4’ cores in three chiplets > Six accelerated compute dies (XCDs) with 38 compute units (CUs), each with 32 KB of L1 cache, 4 MB L2 cache shared across CUs, and 256 MB AMD Infinity Cache™ shared between XCDs and CPUs > 128 GB of HBM3 memory shared coherently between CPUs and GPUs with 5.3 TB/s on-package peak throughput
- sliken 2y agoIndeed. Very similar to the grace + hopper, grace + dual hopper, and grace + dual blackwell.
- j45 2y agoMaybe Apple has figured out something better than a unified memory architecture. It's hard to rule out their ability to create silicon that is a step change.
- danjl 2y agoUMA hurts the GPU too much. Widely parallel processing wants to access memory in bigger chunks than a CPU. If you try to mix access and modification, you lose the benefit of widely parallel processing. Other GPU designers have considered and eschewed unified memory models, to the tune of hundreds of millions in research dollars.
- convolvatron 2y agoI agree that single cache-line fetches are pretty poor for parallel vector units, but supporting the former in an environment designed for the latter doesn't seem to off-putting (the CM-5 did this).
- DonHopkins 2y agoBy the way: Does your user name convolvotron refer to the hardware 3d audio audio processing system originally developed at NASA Ames Research? Such a cool name! And it says just what it does. https://spinoff.nasa.gov/node/8965 https://spinoff.nasa.gov/node/8965 https://spinoff.nasa.gov/sites/default/files/thumbnail0000_20020083312.pdf.jpg https://spinoff.nasa.gov/sites/default/files/thumbnail0000_2... https://pubs.aip.org/asa/jasa/article/92/4_Supplement/2376/730514/The-Convolvotron-Real-time-demonstration-of https://pubs.aip.org/asa/jasa/article/92/4_Supplement/2376/7... Body Electric supported the Convolvotron for visually programming VR simulations with 3D sound: https://news.ycombinator.com/item?id=24266722 https://news.ycombinator.com/item?id=24266722 Did you ever meet (or better yet get a tour of Ames from) the late Ron Reisman, and see the virtual reality, flight simulator, and air traffic control systems his research lab developed? Vertical Motion Simulator: https://www.youtube.com/watch?v=5-lHcv_olkE https://www.youtube.com/watch?v=5-lHcv_olkE Marvin Minsky flies a simulator and wears VR goggles: https://www.youtube.com/watch?v=mOKENF_-z8Y https://www.youtube.com/watch?v=mOKENF_-z8Y
- convolvatron 2y agono. but thank you so much for the references. that's actually really great. I needed a username in the early 90s, I had just finished a paper where we microcoded a CM-2 to support high-throughput convolutions with spatially varying kernels for Hubble image correction (before the launched the eyeglasses mission), and I decided I could be the hero or anti-hero of convolution.
- jack-bodine 2y agoNot mentioned in the article, but another motivation behind this could be that with split a CPU/GPU Apple could try to up-sale on both when purchasing Macs. The prices they charge just to go from 16GB to 32GB of RAM is outrageous ($400 for Macbook pro).
- berbec 2y agoI love the fact you can buy two 16GB/256GB Mac Minis and have cash leftover compared to somone that bought a single 32GB/512GB Mac Mini. Apple's upsells are insane
- lotsofpulp 2y agoI’m seeing 16/256 for $600 and 32/512 for $1,200 on apple.com
- andrewmunsell 2y agoOn the Apple Edu store, it's $499 for the 16/256 and $1079 for the 32/512
- oarsinsync 2y agoHahaha wow I just checked Apple UK and the base 16GB/256GB is £600. 32GB upgrade is +£400, 512GB upgrade is +£200. It should not cost that much! 2x Mac mini M4 16GB/256GB should not cost the same as 1x Mac mini M4 32GB/512GB! Can someone help explain this in a way that isn’t just absolute price gauging of the higher end customer base? Are the components genuinely that much more expensive?
- cma 2y ago> Can someone help explain this in a way that isn’t just absolute price gauging of the higher end customer No, it's the same reason Nvidia has a vastly higher margin on datacenter cards: https://en.wikipedia.org/wiki/Price_discrimination https://en.wikipedia.org/wiki/Price_discrimination
- grecy 2y agoI've always wondered if they'll do something for a true monster Mac Pro.. 128 cores, gobs of memory, etc.
- JonChesterfield 2y agoHackintosh on MI300A?
- marxisttemp 2y agoMy guess is we won’t ever see anything more than what can also fit in a Studio. It’s just not worth the extra R&D for whatever fancy interposer tech they’d need to make a standout Mac Pro
- grecy 2y agoThe strange part is the M MacPro still has the monster 100W power supply from the intel days. Why on earth did they leave that?
- oyashirochama 2y agoIt's a little overbuilt, but a 1000w for a 300w processor with ability to add a lot of accessories isn't terrible actually. only needs a few more PCIe devices to make it meet the 50% minimum to reach peak efficieny on the PSU.
- FireBeyond 2y agoFor your PCI devices and whatever they may draw.
- raydev 2y agoExpansion ports. Arguably, the redesign you're looking for is the Mac Studio.
- raydev 2y agoThe way they've treated the Mac Studio by simply not updating it, letting the MacBook Pro M4 nearly surpass it at the top end of performance, doesn't bode well for the future. Seems like they think Ultras aren't worth the investment, let alone building a true "unleashed" SIP. Apple never says "hey what's the fastest and most powerful thing we can build for X price", they always box themselves in with space or energy constraints, so they never truly compete for the high end. The existing Mac Pro body was their chance to do that, and instead they put something designed for a smaller chassis in there.
- ngcc_hk 2y agoGiven they control both hardware and software, can they have both like efficiency core + power core, with some memory is unified (up to 64 GB) and some dGPU has different memory which they sell against NVIDIA ... Apple has to grow further is hard, but having a T$ firm in front of you with a hardware piece you have is much better than building an Apple Car ...
- dagmx 2y agoNotebookcheck is a poor source of parsing technical information. The actual rumour from Kuo is that they’d move to a chiplet style design where the CPU tile and GPU tile are independent. This is actually in the article as linked. That does not however mean that unified memory would go away. It’s just a new packaging system.
- choilive 2y agoYou can split the CPU and GPU and still have UMA. Splitting CPU/GPU is a packaging and interconnect concern and is not mutually exclusive with UMA.
- lincpa 2y ago[dead]