4 ms·

You're misrepresenting what CPU's do exactly, and opaque term "high-performance workloads" does not help it, either. M-class chips have 256-bit (M4) and 512-bit

by tucnak 1y ago

You're misrepresenting what CPU's do exactly, and opaque term "high-performance workloads" does not help it, either. M-class chips have 256-bit (M4) and 512-bit (M3 Max) memory bus per "socket" options, as high as 1024-bit total in M2 Ultra, which is significantly higher than 64-bit and 128-bit DDR5 buses you get in x86 CPU's. For example, my relatively modern datacenter AMD EPYC 8434PN CPU (based on Zen 4c cores) is a six-channel DDR5, effectively 384-bit at 200 GB/s bidirectional bandwidth. Apple Silicon beats it by a factor of 5x. You can get somewhat better with Turin, but not by much, and at perhaps unreasonable premium. Now, like with everything in life, of course, there's highly-specialised datapaths like AVX-512, but then again these only contribute towards single-threaded performance, & you yourself said that "High-performance workloads are not single-threaded programs." Now, as your compute network grows larger the implementation details of the memory fabric (NUMA, etc.) become more pronounced. Suffice to say the SoC co-packaging of CPU and GPU cores, along with some coprocessors, did wonders for Apple Silicon. Strix Halo exists, but it's not competitive by any stretch of imagination. You could say it's unfair, but then again, AMD MI300A (LGA6096 socket) exists, too! Do we count 20k APU's that only come in eights, bundled up in proprietary Infinity Fabric-based chassis towards "outperforming ARM64 in high-perf workloads"... really? Compute-bound is a far cry from high-performance, where the memory bus, and idiosyncrasies of message-passing are King as number of cores in the compute network continues to grow.

avhception 1y ago
There is also IBMs POWER11, with regard to memory bandwidth :)
- tucnak 1y ago
  It's quite impressive what they were able to achieve with ppc64el in recent years, including Linux support for it, too. Unfortunately, they turned the wrong way with proprietary encryption of memory, which may or may not be deliberate as far as backdoors come and go, but in all honesty so much in it is contingent on IBM's proprietary fabric (OSC or what was it?) implementation for tiered memory anyway. There's similar setups from Samsung, even including fully transparent swapping to NVMe for persistence which is really cool, and hard to match in open source setting. I think their slogan could be "unlimited, coherent, persistent, encrypted high-bandwidth memory is here, and we are the only ones that really have it." Disclaimer: proud owner of thoroughbred OpenPOWER system from Raptor
menaerus 1y ago
> Apple Silicon beats it by a factor of 5x Really, 1TB/s of memory bandwidth to and from system memory? I don't believe it since that's impossible from HW limits PoV - there's no such DRAM that would allow such performance and Apple doesn't design their memory sticks ... It is also no more special with their 512-, 768- or 1024-bit memory interface since this is also not designed by them nor it is exclusively reserved to Apple. Intel has it. AMD has it as well. However, regardless of that, and regardless of the way how you're the one skewing the facts, I would be happy to see the benchmark that shows, for example, a sustained load bandwidth of 1TB/s. Do you have one since I couldn't find it? > You can get somewhat better with Turin High-end Intel/AMD server-grade CPUs can achieve a system memory bandwidth of 600-700GB/s. So not somewhat better but 3x better.
- rowanG077 1y ago
  > Really, 1TB/s of memory bandwidth to and from system memory? 5x is false, it's more like 4x. Apple doesn't use memory sticks, they use on-SoC dram ICs. The M3 Ultra has 8 memory channels at 128-bit per channel for a total of 1024-bit memory bus. It uses LPDDR5-6400 so it has 1024-bit * 6400000000 bits = 819.2 gigabytes per second of memory bandwidth.
  - menaerus 1y ago
    You're deceiving yourself and falling for Apple marketing. Regardless of a stick or SoC memory, which has been the case with pretty much SoC in 2010's (nowadays I have no idea), it is not possible to drive the memory with such high speeds.
    - rowanG077 1y ago
      This is definitely citation needed. I very much expect a combined GPU/CPU/NPU load to saturate the memory channels if necessary. This is not some marketing fluff. The channels are real, the number of RAM ICs are physically there and connected.
      - 1y ago
Rohansi 1y ago
Memory bandwidth is just a marketing term for Apple at this point. Sure, the bus is capable of reaching that bandwidth, but how much can your code actually use? You'd be mistaken if you think the CPU can make use of all that bandwidth, or even the GPU!
- tucnak 1y ago
  It's solely dependent on the workload's memory access patterns. The higher you go in thread count, the more you're constrained by contention, caches, etc. The paper in OP is demonstrating how relatively subtle differences in the memory model are leading to substantial differences in performance on actual hardware. The same as having lots of FLOPS on paper doesn't necessarily mean you'll get to use all that compute, if you're waiting on memory all the time. M-series processors have packaging advantage that is very hard to beat, and indeed, is yet to be beat—in consumer and prosumer segments. See my reply to adjacent comment; hardware is not marketing, and LLM inference stands to witness.
  - Rohansi 1y ago
    > The same as having lots of FLOPS on paper doesn't necessarily mean you'll get to use all that compute, if you're waiting on memory all the time. The opposite case is also possible. You can be compute limited. Or there could be bottlenecks somewhere else. This is definitely the case for Apple Silicon because you will certainly not be able to make use of all of the memory bandwidth from the CPU or GPU. As always, benchmark instead of looking at raw hardware specifications.
- inkyoto 1y ago
  > […] but how much can your code actually use? All of it, and it is transparent to the code. The correct question is «how much data does the code transfer?» Whether you are scanning large string ropes for a lone character or multiplying huge matrices, no manual code optimisation is required.
  - tucnak 1y ago
    Are you well-read enough into the platform so that you can attest to it requiring no manual code optimisation for high-performance datapaths? I'm only familiar with Apple Silicon-specific code in llama.cpp, and not really familiar with either Accelerate[0] or MLX[1] specifically. Have they really cracked it at homogenous computing so that you could use a single description of computation, and have it emit efficient code for whatever target in the SoC? Or are you merely referring to the full memory capacity/bandwidth being available to CPU in normal operation? [0]: https://developer.apple.com/documentation/accelerate https://developer.apple.com/documentation/accelerate [1]: https://ml-explore.github.io/mlx/build/html/usage/quick_start.html https://ml-explore.github.io/mlx/build/html/usage/quick_star...

by tucnak 1y ago

avhception 1y ago

There is also IBMs POWER11, with regard to memory bandwidth :)

tucnak 1y ago
It's quite impressive what they were able to achieve with ppc64el in recent years, including Linux support for it, too. Unfortunately, they turned the wrong way with proprietary encryption of memory, which may or may not be deliberate as far as backdoors come and go, but in all honesty so much in it is contingent on IBM's proprietary fabric (OSC or what was it?) implementation for tiered memory anyway. There's similar setups from Samsung, even including fully transparent swapping to NVMe for persistence which is really cool, and hard to match in open source setting. I think their slogan could be "unlimited, coherent, persistent, encrypted high-bandwidth memory is here, and we are the only ones that really have it." Disclaimer: proud owner of thoroughbred OpenPOWER system from Raptor

menaerus 1y ago

> Apple Silicon beats it by a factor of 5x Really, 1TB/s of memory bandwidth to and from system memory? I don't believe it since that's impossible from HW limits PoV - there's no such DRAM that would allow such performance and Apple doesn't design their memory sticks ... It is also no more special with their 512-, 768- or 1024-bit memory interface since this is also not designed by them nor it is exclusively reserved to Apple. Intel has it. AMD has it as well. However, regardless of that, and regardless of the way how you're the one skewing the facts, I would be happy to see the benchmark that shows, for example, a sustained load bandwidth of 1TB/s. Do you have one since I couldn't find it? > You can get somewhat better with Turin High-end Intel/AMD server-grade CPUs can achieve a system memory bandwidth of 600-700GB/s. So not somewhat better but 3x better.

rowanG077 1y ago
> Really, 1TB/s of memory bandwidth to and from system memory? 5x is false, it's more like 4x. Apple doesn't use memory sticks, they use on-SoC dram ICs. The M3 Ultra has 8 memory channels at 128-bit per channel for a total of 1024-bit memory bus. It uses LPDDR5-6400 so it has 1024-bit * 6400000000 bits = 819.2 gigabytes per second of memory bandwidth.
- menaerus 1y ago
  You're deceiving yourself and falling for Apple marketing. Regardless of a stick or SoC memory, which has been the case with pretty much SoC in 2010's (nowadays I have no idea), it is not possible to drive the memory with such high speeds.
  - rowanG077 1y ago
    This is definitely citation needed. I very much expect a combined GPU/CPU/NPU load to saturate the memory channels if necessary. This is not some marketing fluff. The channels are real, the number of RAM ICs are physically there and connected.
    - 1y ago

Rohansi 1y ago

Memory bandwidth is just a marketing term for Apple at this point. Sure, the bus is capable of reaching that bandwidth, but how much can your code actually use? You'd be mistaken if you think the CPU can make use of all that bandwidth, or even the GPU!

tucnak 1y ago
It's solely dependent on the workload's memory access patterns. The higher you go in thread count, the more you're constrained by contention, caches, etc. The paper in OP is demonstrating how relatively subtle differences in the memory model are leading to substantial differences in performance on actual hardware. The same as having lots of FLOPS on paper doesn't necessarily mean you'll get to use all that compute, if you're waiting on memory all the time. M-series processors have packaging advantage that is very hard to beat, and indeed, is yet to be beat—in consumer and prosumer segments. See my reply to adjacent comment; hardware is not marketing, and LLM inference stands to witness.
- Rohansi 1y ago
  > The same as having lots of FLOPS on paper doesn't necessarily mean you'll get to use all that compute, if you're waiting on memory all the time. The opposite case is also possible. You can be compute limited. Or there could be bottlenecks somewhere else. This is definitely the case for Apple Silicon because you will certainly not be able to make use of all of the memory bandwidth from the CPU or GPU. As always, benchmark instead of looking at raw hardware specifications.
inkyoto 1y ago
> […] but how much can your code actually use? All of it, and it is transparent to the code. The correct question is «how much data does the code transfer?» Whether you are scanning large string ropes for a lone character or multiplying huge matrices, no manual code optimisation is required.
- tucnak 1y ago
  Are you well-read enough into the platform so that you can attest to it requiring no manual code optimisation for high-performance datapaths? I'm only familiar with Apple Silicon-specific code in llama.cpp, and not really familiar with either Accelerate[0] or MLX[1] specifically. Have they really cracked it at homogenous computing so that you could use a single description of computation, and have it emit efficient code for whatever target in the SoC? Or are you merely referring to the full memory capacity/bandwidth being available to CPU in normal operation? [0]: https://developer.apple.com/documentation/accelerate https://developer.apple.com/documentation/accelerate [1]: https://ml-explore.github.io/mlx/build/html/usage/quick_start.html https://ml-explore.github.io/mlx/build/html/usage/quick_star...