4 ms·

"You'd need a fairly drastic shift in the memory architecture of CPU". You mean like selling laptops (at 400GB/sec) and desktops (at 800GB/sec) with much impro

by sliken 4y ago

"You'd need a fairly drastic shift in the memory architecture of CPU". You mean like selling laptops (at 400GB/sec) and desktops (at 800GB/sec) with much improved memory systems. I don't want to give up SO-DIMMs for a few mm thinner laptop, but going from the intel/amd standard 70GB/sec to 400GB/sec is a pretty big incentive.

kllrnohj 4y ago
M1's CPUs can't achieve that rate fwiw. It's also generic LPDDR5, not actually anything particularly new or exciting. The 800GB/s isn't to a single compute unit, either, it's 2x 400GB/s controllers. Regardless, no that's not the drastic shift necessary. It's not even a shift at all. 4, 6, and 8 channel memory architectures have all existed for a while now in data center SKUs. It hasn't changed the desire for GPU compute in the data center
- sliken 4y ago
  A comment here on benchmarking the M1 managed to get 360GB/sec observed out of a 400GB/sec memory system, so I'd consider that pretty good and certainly a better percentage of peak than I see out of the x86-64 systems I benchmarked. Well the 4, 6, and 8 channel memory architectures have been dramatically slower than the Apple 800GB/sec with dramatically less memory transactions in flight (1 per channel) than Apple's 800GB/sec with 64 channels. The push for Apple like memory systems with HBM from Intel, AMD, Fujitsu and others does show that there's a need for more compute bandwidth and Fujitsu in particular has shown that a healthy CPU with a GPU like memory system can get great performance on real world codes without the hassle of rewriting for CUDA. With all that said, yes there's a healthy demand for GPUs and codes that run best on GPUs. However I find it refreshing that at last one company is providing significantly improved memory systems in laptops and small desktops.
  - kllrnohj 4y ago
    > A comment here on benchmarking the M1 managed to get 360GB/sec observed out of a 400GB/sec memory system, so I'd consider that pretty good and certainly a better percentage of peak than I see out of the x86-64 systems I benchmarked. "the M1 Max isn’t able to fully saturate the SoC bandwidth from just the CPU side; From a single core perspective [..] it’s able to stress the memory fabric to up to 102GB/s. [..] Adding a third thread there’s a bit of an imbalance across the clusters, DRAM bandwidth goes to 204GB/s, but a fourth thread lands us at 224GB/s and this appears to be the limit on the SoC fabric that the CPUs are able to achieve" https://www.anandtech.com/show/17024/apple-m1-max-performance-review/2 https://www.anandtech.com/show/17024/apple-m1-max-performanc... So peak CPU bandwidth on the M1 Max is 224GB/s. Which is really good for a CPU to be sure, but we're still talking bandwidth numbers far below what you'd need to eliminate the GPU. > Well the 4, 6, and 8 channel memory architectures have been dramatically slower than the Apple 800GB/sec with dramatically less memory transactions in flight (1 per channel) Huh? I guarantee you Xeons and Epycs are not limited to 1 transaction in flight, nor is Graviton2. The AMD APUs in things like the steam deck, PS4, and PS5 (448GB/s fwiw) are also definitely not limited like that, either. > than Apple's 800GB/sec with 64 channels M1 ultra is 32 channels. Well, it's really 2x 16 channels since it's a NUMA configuration, like a dual-socket system. > The push for Apple like memory systems with HBM from Intel, AMD, Fujitsu lol? Apple's memory system isn't like HBM. It's more like a traditional GPU's but with GDDR swapped out for LPDDR. M1 Max is 512-bit memory bus, just like many GPUs have done over the years. 512-bit bus and it can only hit 400GB/s shows you just how much less bandwidth LPDDR has vs. GDDR, too. AMD & NVidia are hitting 1TB/s with 384-bit buses these days. Which is also what the HBM2e Xeon's are claiming.
    - sliken 4y ago
      > "the M1 Max isn’t able to fully saturate the SoC bandwidth from just the CPU side Sure, wouldn't want to starve the GPU when the CPU is busy. Not sure if the AMX has it's own connection or shares the CPU complex bandwidth. > I guarantee you Xeons and Epycs are not limited to 1 transaction in flight, nor is Graviton2 Right, I said 1 per channel, not 1. I've tested this on Xeons and Epycs, but not genoa yet. Generally maximum random throughput tends to be with 16 or so cache misses per socket with 8 channels. From what I can tell about half the latency is cache misses in L1, L2, and L3. Then you queue in the memory controller and wait for whatever memory channel (or two depending on config). By keeping 16 in the queue as soon as one of the 8 channels free you have another request pending for that channel, at least most of the time. The Genoa looks particularly promising on this front, they upgraded from 8 channels to 12, but the DDR5 dimms actually provide two channels per dimm, so you end up with 24 narrow (32 bit) DDR5 channels. I've already seen results showing that Genoa scales better to high core counts than the zen3 Epyycs, which makes sense with 24 memory requests in flight instead of 8. > M1 ultra is 32 channels. Well, it's really 2x 16 channels since it's a NUMA configuration, like a dual-socket system. Heh, well monolithic chips are on the way out. Even the zen2 epycs used multiple pieces of silicon (chiplets) and the new Intel Xeon/Sapphire Rapids do the same. Don't see a particular difference. Even the desktop Ryzens have multiple pieces of silicon (one IOD and one CPU chiplet at the minimum). In any case the Apple M1 Max memory system matches the Genoa and beats the newest Intel Sapphire Rapids with a single piece of silicon, and wins handily doubles that with two pieces of silicon, unless you wait for the unreleased HBM flavors. Compared to 100% of Intel/AMD laptops and probably 98% of desktops that apple memory system is pretty amazing. Sure there are threadrippers (4 channel), threadripper pro (8 channel), and some similar Intel options, but all are expensive, low volume, and desktop/workstation only.
      - kllrnohj 4y ago
        > Sure, wouldn't want to starve the GPU when the CPU is busy. Ok, then surely you won't take issue with me describing my x86 desktop as having 1.2TB/s of memory bandwidth? It just happens to be in a NUMA configuration and after all, wouldn't want to starve the GPU when the CPU is busy... > Don't see a particular difference. You don't see a difference between 1 memory controller and 2 memory controllers? Compare Zen1 TR/epyc vs. zen2 TR/epyc. Hint: m1 ultra is like Zen1, and probably ain't the future Oh yeah and Intel did this, too, with the 56 core xeons a couple years ago. And, like AMD, also quickly moved away from it.

by sliken 4y ago

kllrnohj 4y ago

M1's CPUs can't achieve that rate fwiw. It's also generic LPDDR5, not actually anything particularly new or exciting. The 800GB/s isn't to a single compute unit, either, it's 2x 400GB/s controllers. Regardless, no that's not the drastic shift necessary. It's not even a shift at all. 4, 6, and 8 channel memory architectures have all existed for a while now in data center SKUs. It hasn't changed the desire for GPU compute in the data center

sliken 4y ago
A comment here on benchmarking the M1 managed to get 360GB/sec observed out of a 400GB/sec memory system, so I'd consider that pretty good and certainly a better percentage of peak than I see out of the x86-64 systems I benchmarked. Well the 4, 6, and 8 channel memory architectures have been dramatically slower than the Apple 800GB/sec with dramatically less memory transactions in flight (1 per channel) than Apple's 800GB/sec with 64 channels. The push for Apple like memory systems with HBM from Intel, AMD, Fujitsu and others does show that there's a need for more compute bandwidth and Fujitsu in particular has shown that a healthy CPU with a GPU like memory system can get great performance on real world codes without the hassle of rewriting for CUDA. With all that said, yes there's a healthy demand for GPUs and codes that run best on GPUs. However I find it refreshing that at last one company is providing significantly improved memory systems in laptops and small desktops.
- kllrnohj 4y ago
  > A comment here on benchmarking the M1 managed to get 360GB/sec observed out of a 400GB/sec memory system, so I'd consider that pretty good and certainly a better percentage of peak than I see out of the x86-64 systems I benchmarked. "the M1 Max isn’t able to fully saturate the SoC bandwidth from just the CPU side; From a single core perspective [..] it’s able to stress the memory fabric to up to 102GB/s. [..] Adding a third thread there’s a bit of an imbalance across the clusters, DRAM bandwidth goes to 204GB/s, but a fourth thread lands us at 224GB/s and this appears to be the limit on the SoC fabric that the CPUs are able to achieve" https://www.anandtech.com/show/17024/apple-m1-max-performance-review/2 https://www.anandtech.com/show/17024/apple-m1-max-performanc... So peak CPU bandwidth on the M1 Max is 224GB/s. Which is really good for a CPU to be sure, but we're still talking bandwidth numbers far below what you'd need to eliminate the GPU. > Well the 4, 6, and 8 channel memory architectures have been dramatically slower than the Apple 800GB/sec with dramatically less memory transactions in flight (1 per channel) Huh? I guarantee you Xeons and Epycs are not limited to 1 transaction in flight, nor is Graviton2. The AMD APUs in things like the steam deck, PS4, and PS5 (448GB/s fwiw) are also definitely not limited like that, either. > than Apple's 800GB/sec with 64 channels M1 ultra is 32 channels. Well, it's really 2x 16 channels since it's a NUMA configuration, like a dual-socket system. > The push for Apple like memory systems with HBM from Intel, AMD, Fujitsu lol? Apple's memory system isn't like HBM. It's more like a traditional GPU's but with GDDR swapped out for LPDDR. M1 Max is 512-bit memory bus, just like many GPUs have done over the years. 512-bit bus and it can only hit 400GB/s shows you just how much less bandwidth LPDDR has vs. GDDR, too. AMD & NVidia are hitting 1TB/s with 384-bit buses these days. Which is also what the HBM2e Xeon's are claiming.
  - sliken 4y ago
    > "the M1 Max isn’t able to fully saturate the SoC bandwidth from just the CPU side Sure, wouldn't want to starve the GPU when the CPU is busy. Not sure if the AMX has it's own connection or shares the CPU complex bandwidth. > I guarantee you Xeons and Epycs are not limited to 1 transaction in flight, nor is Graviton2 Right, I said 1 per channel, not 1. I've tested this on Xeons and Epycs, but not genoa yet. Generally maximum random throughput tends to be with 16 or so cache misses per socket with 8 channels. From what I can tell about half the latency is cache misses in L1, L2, and L3. Then you queue in the memory controller and wait for whatever memory channel (or two depending on config). By keeping 16 in the queue as soon as one of the 8 channels free you have another request pending for that channel, at least most of the time. The Genoa looks particularly promising on this front, they upgraded from 8 channels to 12, but the DDR5 dimms actually provide two channels per dimm, so you end up with 24 narrow (32 bit) DDR5 channels. I've already seen results showing that Genoa scales better to high core counts than the zen3 Epyycs, which makes sense with 24 memory requests in flight instead of 8. > M1 ultra is 32 channels. Well, it's really 2x 16 channels since it's a NUMA configuration, like a dual-socket system. Heh, well monolithic chips are on the way out. Even the zen2 epycs used multiple pieces of silicon (chiplets) and the new Intel Xeon/Sapphire Rapids do the same. Don't see a particular difference. Even the desktop Ryzens have multiple pieces of silicon (one IOD and one CPU chiplet at the minimum). In any case the Apple M1 Max memory system matches the Genoa and beats the newest Intel Sapphire Rapids with a single piece of silicon, and wins handily doubles that with two pieces of silicon, unless you wait for the unreleased HBM flavors. Compared to 100% of Intel/AMD laptops and probably 98% of desktops that apple memory system is pretty amazing. Sure there are threadrippers (4 channel), threadripper pro (8 channel), and some similar Intel options, but all are expensive, low volume, and desktop/workstation only.
    - kllrnohj 4y ago
      > Sure, wouldn't want to starve the GPU when the CPU is busy. Ok, then surely you won't take issue with me describing my x86 desktop as having 1.2TB/s of memory bandwidth? It just happens to be in a NUMA configuration and after all, wouldn't want to starve the GPU when the CPU is busy... > Don't see a particular difference. You don't see a difference between 1 memory controller and 2 memory controllers? Compare Zen1 TR/epyc vs. zen2 TR/epyc. Hint: m1 ultra is like Zen1, and probably ain't the future Oh yeah and Intel did this, too, with the 56 core xeons a couple years ago. And, like AMD, also quickly moved away from it.