5 ms·
Apple is using LPDDR5 for M3. The bandwidth doesn't come from unified memory - it comes from using many channels. You could get the same bandwidth or more with
by Rohansi 2y ago
Apple is using LPDDR5 for M3. The bandwidth doesn't come from unified memory - it comes from using many channels. You could get the same bandwidth or more with normal DDR5 modules if you could use 8 or more channels, but in the PC space you don't usually see more than 2 or 4 channels (only common for servers).
Unrelated but unified memory is a strange buzzword being used by Apple. Their memory is no different than other computers. In fact, every computer without a discrete GPU uses a unified memory model these days!
- manaskarekar 2y agoYes, it's just easier to call it that without having to sprinkle asterisks at each mention of it :) And yes, the impressive part is that this kind of bandwidth is hard to get on laptops. I suppose I should have been a bit more specific in my remark.
- binary132 2y agoI read all that marketing stuff and my brain just sees APU. I guess at some level, that’s just marketing stuff too, but it’s not a new idea.
- sroussey 2y agoEh… not quite. Maybe on an Instinct. Unified memory means the CPU and CPU means they can do zero copy to use the same memory buffer. Many integrated graphics segregate the memory into CPU owned and GPU owned, so that even if data is on the same DIMM, a copy still needs to be performed for one side to use what the other side already has. This means that the drivers, etc, all have to understand the unified memory model, etc. it’s not just hardware sharing DIMMs.
- binary132 2y agoI was under the impression PS4’s APU implemented unified memory, and it was even referred to by that name[1]. APUs with shared everything are not a new concept, they are actually older than programmable graphics coprocessors… https://www.heise.de/news/Gamescom-Playstation-4-bietet-Unified-Memory-Xbox-One-nicht-1939716.html https://www.heise.de/news/Gamescom-Playstation-4-bietet-Unif...
- sunshowers 2y agoI believe that at least on Linux you get zero-copy these days. https://www.phoronix.com/news/AMD-AOMP-19.0-2-Compiler https://www.phoronix.com/news/AMD-AOMP-19.0-2-Compiler
- sliken 2y agoThe new idea is having 512 bit wide memory instead of PC limitation of 128 bit wide. Normal CPU cores running normal codes are not particularly bandwidth limited. However APUs/iGPUs are severely bandwidth limited, thus the huge number of slow iGPUs that are fine for browsing but terrible for anything more intensive. So apple manages decent GPU performance, a tiny package, and great battery life. It's much harder on the PC side because every laptop/desktop chip from Intel and AMD use a 128 bit memory bus. You have to take a huge step up in price, power, and size with something like a thread ripper, xeon, or epyc to get more than 128 bit wide memory, none of which are available in a laptop or mac mini size SFF.
- reliabilityguy 2y ago> instead of PC limitation of 128 bit wide Memory interface width of modern CPUs is 64-bit (DDR4) and 32+32 (DDR5). No CPU uses 128b memory bus as it results in overfetch of data, i.e., 128B per access, or two cache lines. AFAIK Apple uses 128B cache lines, so they can do much better design and customization of memory subsystem as they do not have to use DIMMs -- they simply solder DRAM to the motherboard, hence memory interface is whatever they want.
- sliken 2y ago> Memory interface width of modern CPUs is 64-bit (DDR4) and 32+32 (DDR5). Sure, per channel. PCs have 2x64 bit or 4x32 bit memory channels. Not sure I get your point, yes PCs have 64 bit cache lines and apple uses 128. I wouldn't expect any noticeable difference because of this. Generally cache miss is sent to a single memory channel and result in a wait of 50-100ns, then you get 4 or 8 bytes per cycle at whatever memory clock speed you have. So apple gets twice the bytes per cache line miss, but the value of those extra bytes is low in most cases. Other bigger differences is that apple has a larger page size (16KB vs 4KB) and arm supports a looser memory model, which makes it easier to reach a large fraction of peak memory bandwidth. However, I don't see any relationship between Apple and PCs as far as DIMMS. Both Apple and PCs can (and do) solder dram chips directly to the motherboard, normally on thin/light laptops. The big difference between Apple and PC is that apple supports 128, 256, and 512 bit wide memory on laptops and 1024 bit on the studio (a bit bigger than most SFFs). To get more than 128 bits with a PC that means no laptops, no SFFs, generally large workstations with Xeon, Threadrippers, or Epyc with substantial airflow and power requirements
- oDot 2y agoIsn't unified memory* a crucial part in avoiding signal integrity problems? Servers do have many channels but they run relatively slower memory * Specifically, it being on-die
- wtallis 2y ago"Unified memory" doesn't really imply anything about the memory being located on-package, just that it's a shared pool that the CPU, GPU, etc. all have fast access to. Also, DRAM is never on-die. On-package, yes, for Apple's SoCs and various other products throughout the industry, but DRAM manufacturing happens in entirely different fabs than those used for logic chips.
- kube-system 2y agoSystem memory DRAM never is, but sometimes DRAM is technically included on CPU dies as a cache https://en.wikipedia.org/wiki/EDRAM https://en.wikipedia.org/wiki/EDRAM
- wtallis 2y agoIt's mostly an IBM thing. In the consumer space, it's been in game consoles with IBM-fabbed chips. Intel's use of eDRAM was on a separate die (there was a lot that was odd about those parts).
- Tepix 2y agoFor comparison, a Threadripper Pro 5000 workstation with 8x DDR4 3200 has 204.8GB/s of memory bandwidth. The Threadripper Pro 7000 with DDR5-5200 can achieve 325GB/s. And no, manaskarekar, the M4 Max does 546 GB/s not GBps (which would be 8x less!).
- wtallis 2y ago> And no, manaskarekar, the M4 Max does 546 GB/s not GBps (which would be 8x less!). GB/s and GBps mean the same thing, though GB/s is the more common way to express it. Gb/s and Gbps are the units that are 8x less: bits vs Bytes.
- deleted 2y ago[deleted]
- hmottestad 2y agoThanks for the numbers. Someone here on hackernews got me convinced that a Threadripper would be a better investment for inference than a MacBook Pro with a M3 Max.
- leptons 2y agoB = Bytes, b = bits. GB/s is the same thing as GBps The "ps" means "per second"
- rbanffy 2y ago> (only common for servers). On PC desktops I always recommend getting a mid-range tower server precisely for that reason. My oldest one is about 8 years old and only now it's showing signs of age (as in not being faster than the average laptop).
- throwaway48476 2y agoHigh end servers now have 12 ddr5 channels.
- sliken 2y agoYes, you could buy a brand new (announced weeks ago) AMD Turin. 12 channels of DDR5-6000, $11,048 and 320 watts (for the CPU) and get 576GB/sec peak. Or you could buy a M3 max laptop for $4k, get 10+ hour battery life, have it fit in a thin/light laptop, and still get 546GB/sec. However those are peak numbers. Apple uses longer cache lines (double), large page sizes (quadruple), and a looser memory model. Generally I'd expect nearly every memory bandwidth measure to win on Apple over AMD's turin.
- Rohansi 2y agoAnandTech did bandwidth benchmarks for the M1 Max and was only able to utilize about half of it from the CPU, and the GPU used even less in 3D workloads because it wasn't bandwidth limited. It's not all about bandwidth. https://www.anandtech.com/show/17024/apple-m1-max-performance-review/2 https://www.anandtech.com/show/17024/apple-m1-max-performanc...
- sliken 2y agoIndeed. RIP Anandtech. I've seen bandwidth tests since then that showed similar for newer generations, but not the m4. Not sure if the common LLM tools on mac can use CPU (vector instructions), AMX, and Neural engine in parallel to make use of the full bandwidth.
- throwaway48476 2y agoCXL memory is also a thing.
- janwas 2y agoFWIW I ran a quick test of gemma.cpp on M3 Pro with 8 threads. Similar PaliGemma inference speed to an older AMD (Rome or Milan) with 8 threads. But the AMD has more cores than that, and more headroom :)
- sunshowers 2y agoYeah memory bandwidth is one of the really unfortunate things about the consumer stuff. Even the 9950x/7950x, which are comfortably workstation-level in terms of compute, are bound by their 2 channel limits. The other day I was pricing out a basic Threadripper setup with a 7960x (not just for this reason but also for more PCIe lanes), and it would cost around $3000 -- somewhat out of my budget. This is one of the reasons the "3D vcache" stuff with the giant L3 cache is so effective.
- astrange 2y ago> In fact, every computer without a discrete GPU uses a unified memory model these days! On PCs some other hardware (notably the SSD) comes with its own memory. But here it's shared with the main DRAM too. This is not necessarily a performance improvement, it can avoid copies but also means less is available to the CPU.
- kalleboo 2y agoDRAM-less NVMe (utilizing HMB) is also common on PCs, but it's seen as a slower budget alternative rather than a good thing.