7 ms·

192GB ram does seem like enough for a large fraction of the potential market, especially @ 800GB/sec which makes it much more usable than similar amounts of ram

by sliken 3y ago

192GB ram does seem like enough for a large fraction of the potential market, especially @ 800GB/sec which makes it much more usable than similar amounts of ram on Intel/AMD desktops at 1/8th the bandwidth.

StillBored 3y ago
Except that my fairly modest upper midrange desktop has ~1.2TB/sec of memory bandwidth... Cause, I just added the GPU and system RAM bandwidth numbers together. Which is what needs to be kept in mind with much of this. Yes that is a lot of memory bandwidth and its hella useful for some subset of users, but its shared, and largely pointless for a lot of CPU bound tasks. But OTOH, may not be enough for many GPU bound ones. It also assumes that pretty much every other CPU manufacture on the planet are idiots for optimizing for latency and putting in large caches to compensate (aka the desktop parts from AMD/intel have only _two_ channels, vs the 8+ in the server/workstation parts) and price discriminating for the parts that have more CPU bandwidth. AKA, you can get amd machines in the same ballpark (or possibly faster depending on how fast you can get 24 channels of DDR5 to run). So, I'm not saying which is better because its likely workload dependent, but to claim its a blanket insurmountable advantage is questionable. Particularly since the price ranges we are talking about a similar machine is probably a 64 core threadripper plus a fat nvidia GPU or four and the shear core count and raw GPU compute is probably a win in most workloads.
- USB5 3y ago
  I just built a workstation with a 32 core Threadripper Pro, 128 GB ECC RAM, Thunderbolt 4, and an RTX 4090 for $6,500.
  - sliken 3y ago
    Likely better on any workloads that are GPU heavy and fit in 24GB of vram. Two data points from Apple: a) M2 Ultra 24 cores, 128GB ram, 2TB storage = $5,200 b) M2 ultra 24 cores, 192GB ram, 4GB storage = $6,600 Likely 1/4th the size, 1/4th the power consumption, and 4x the ram bandwidth. Have you by chance played with any LLMs? Just saw a post that someone managed 5 tokens/sec with the llama 65B model.
    - USB5 3y ago
      I don't care about playing with LLMs, so this is simply meaningless to me.
      - kamranjon 3y ago
        But for many people training or running inference on large models is important and the apple approach actually makes that feasible in a desktop solution which is pretty awesome.
    - deleted 3y ago
      [deleted]
- sliken 3y ago
  Sure, but what if a normal C code needs more bandwidth? Or if a GPU code needs more than 12-16GB of memory (normal cards) or 24GB (if you get a 4090)? What I like about the apple approach is that low end laptops/desktops get 100GB/sec. Pay another $500 get 200GB/sec. Pay another $500 get 400GB/sec. Pay another $1000 get 800GB/sec and still fits in a small desktop. On the PC side with AMD/Intel you get the same memory bandwidth for the low, medium, and high end chips. Until you upgrade to a threadripper, which is a 280 watt chip, on an expensive motherboard, usually in a rather large PC case and makes the mac studio look cheap.
  - deleted 3y ago
    [deleted]
  - kcb 3y ago
    > Sure, but what if a normal C code needs more bandwidth? https://www.anandtech.com/show/17024/apple-m1-max-performance-review/2 https://www.anandtech.com/show/17024/apple-m1-max-performanc...
    - wtallis 3y ago
      Putting things in context, the 3-thread measured memory bandwidth measured there on the M1 Max is approximately equal to the maximum theoretical memory bandwidth available across all channels and chiplets of a current Ryzen Threadripper PRO. If the M1 Ultra doubles the achievable memory bandwidth by virtue of having twice as many CPU clusters, then an 8-thread test should still be able to match AMD's latest EPYC processors with 12-channel DDR5. (I don't know how well AMD's current processors do with utilizing the socket's full DRAM bandwidth from a limited number of chiplets, but I wouldn't be surprised if it's a more severe limitation than what M1 Max/Ultra show with their CPU clusters. It looks like only the 12-chiplet EPYC processors actually use all the links from the IO die to the CPU chiplets.) So the inability to use all the DRAM bandwidth from the CPU cores, while perhaps disappointing, isn't exactly a weakness for Apple's processors compared to the competition.
      - sliken 3y ago
adgjlsfhk1 3y ago
This is competing with servers though, and Epyc 9004 series has 460 GB/sec (and up to 6TB of ram per socket). Apple still gives you faster connection, but I feel like servers below 256GB ram are pretty rare these days.
221qqwe 3y ago
> much more usable than similar amounts of ram People keep repeating this but how does higher bandwidth (probably not 8x higher though) compensate for a lower amount of RAM? It's not quite as silly as the people saying that 8GB in the base config 'feels' much faster than 8GB on a PC cause the drive/swap are "so fast" but still..
- lambdasquirrel 3y ago
  Maybe it adds another step to the memory mountain? Folks who use these kinds of workstations might think of in-package RAM as just the next level of cache, if someone goes ahead and makes a card with comparatively slower memory card slots.
- dathinab 3y ago
  The DDR5 bandwidth is ~51.2 GiB/s per "channel". With consumer desktop CPUs having 2 "channels" (~102.4GiB/s). And prosumer desktop/workstation CPUs (e.g. Threadripper) having 4 "channels" (~204.8GiB/s). While apple is claiming 819.2GiB/s. That is _max throughput is 4x/8x more_ (depending if you compare it to prosumer (fair comparison) or consumer (unfair comparison for a $7k system) hardware. Now the _max_ part is important, Apple mainly reaches this by having more bandwidth, i.e. parallelism. Mainly (oversimplified) the per "CPU memory channel" bandwidth for x86 is 64bit (for 51.2GiB/s) so the M1 Ultra is roughly comparable to having 16 "CPU memory channels" instead of 8 (prosumer) or 4 (consumer). Some applications can take advantage of this nicely and will scale potentially even to 4x/8x speed, most probably will not some might even have neglible improvements. But applications which use a lot of RAM (as much as they can get) and most of the RAM they use is "warm" (e.g. doesn't just lie around with little access but isn't super "hot", i.e. highly contented either) will profit quite a lot. On the other hand applications which use little RAM but the same small RAM region very heavily contented likely will hardly at all profit and will likely run faster on overclocked RAM on consumer systems. Luckily for apple most the typical use case they sell their pro desktop model for belong more/mostly in the first category. Additionally if you can run Linux on this system they might become _very_ interesting for some scientific applications for some users. I mean even e.g. Zen 4 EPYC CPU only have 12 "CPU memory channels" not 16 and it's much easier to put a desktop box "somewhere" then a server unit. Side note: I say "CPU channel" in quotes because while it tends to be the marketing term things are more complicated in practice, e.g. in general DDR5 is splitting the 64bit channels into 32bit sub-channels, and just listing the channel width and throughput is still not painting the whole picture at all, e.g. the latency also matters for some applications (hence why OC can make sense) etc. EDIT: Correction: The Threadripper PRO models have 8 "CPU memory channels" so it's just 2x on a fair comparison.

by sliken 3y ago

StillBored 3y ago

Except that my fairly modest upper midrange desktop has ~1.2TB/sec of memory bandwidth... Cause, I just added the GPU and system RAM bandwidth numbers together. Which is what needs to be kept in mind with much of this. Yes that is a lot of memory bandwidth and its hella useful for some subset of users, but its shared, and largely pointless for a lot of CPU bound tasks. But OTOH, may not be enough for many GPU bound ones. It also assumes that pretty much every other CPU manufacture on the planet are idiots for optimizing for latency and putting in large caches to compensate (aka the desktop parts from AMD/intel have only _two_ channels, vs the 8+ in the server/workstation parts) and price discriminating for the parts that have more CPU bandwidth. AKA, you can get amd machines in the same ballpark (or possibly faster depending on how fast you can get 24 channels of DDR5 to run). So, I'm not saying which is better because its likely workload dependent, but to claim its a blanket insurmountable advantage is questionable. Particularly since the price ranges we are talking about a similar machine is probably a 64 core threadripper plus a fat nvidia GPU or four and the shear core count and raw GPU compute is probably a win in most workloads.

USB5 3y ago
I just built a workstation with a 32 core Threadripper Pro, 128 GB ECC RAM, Thunderbolt 4, and an RTX 4090 for $6,500.
- sliken 3y ago
  Likely better on any workloads that are GPU heavy and fit in 24GB of vram. Two data points from Apple: a) M2 Ultra 24 cores, 128GB ram, 2TB storage = $5,200 b) M2 ultra 24 cores, 192GB ram, 4GB storage = $6,600 Likely 1/4th the size, 1/4th the power consumption, and 4x the ram bandwidth. Have you by chance played with any LLMs? Just saw a post that someone managed 5 tokens/sec with the llama 65B model.
  - USB5 3y ago
    I don't care about playing with LLMs, so this is simply meaningless to me.
    - kamranjon 3y ago
      But for many people training or running inference on large models is important and the apple approach actually makes that feasible in a desktop solution which is pretty awesome.
  - deleted 3y ago
    [deleted]
sliken 3y ago
Sure, but what if a normal C code needs more bandwidth? Or if a GPU code needs more than 12-16GB of memory (normal cards) or 24GB (if you get a 4090)? What I like about the apple approach is that low end laptops/desktops get 100GB/sec. Pay another $500 get 200GB/sec. Pay another $500 get 400GB/sec. Pay another $1000 get 800GB/sec and still fits in a small desktop. On the PC side with AMD/Intel you get the same memory bandwidth for the low, medium, and high end chips. Until you upgrade to a threadripper, which is a 280 watt chip, on an expensive motherboard, usually in a rather large PC case and makes the mac studio look cheap.
- deleted 3y ago
  [deleted]
- kcb 3y ago
  > Sure, but what if a normal C code needs more bandwidth? https://www.anandtech.com/show/17024/apple-m1-max-performance-review/2 https://www.anandtech.com/show/17024/apple-m1-max-performanc...
  - wtallis 3y ago
    Putting things in context, the 3-thread measured memory bandwidth measured there on the M1 Max is approximately equal to the maximum theoretical memory bandwidth available across all channels and chiplets of a current Ryzen Threadripper PRO. If the M1 Ultra doubles the achievable memory bandwidth by virtue of having twice as many CPU clusters, then an 8-thread test should still be able to match AMD's latest EPYC processors with 12-channel DDR5. (I don't know how well AMD's current processors do with utilizing the socket's full DRAM bandwidth from a limited number of chiplets, but I wouldn't be surprised if it's a more severe limitation than what M1 Max/Ultra show with their CPU clusters. It looks like only the 12-chiplet EPYC processors actually use all the links from the IO die to the CPU chiplets.) So the inability to use all the DRAM bandwidth from the CPU cores, while perhaps disappointing, isn't exactly a weakness for Apple's processors compared to the competition.
    - sliken 3y ago

adgjlsfhk1 3y ago

This is competing with servers though, and Epyc 9004 series has 460 GB/sec (and up to 6TB of ram per socket). Apple still gives you faster connection, but I feel like servers below 256GB ram are pretty rare these days.

221qqwe 3y ago

> much more usable than similar amounts of ram People keep repeating this but how does higher bandwidth (probably not 8x higher though) compensate for a lower amount of RAM? It's not quite as silly as the people saying that 8GB in the base config 'feels' much faster than 8GB on a PC cause the drive/swap are "so fast" but still..

lambdasquirrel 3y ago
Maybe it adds another step to the memory mountain? Folks who use these kinds of workstations might think of in-package RAM as just the next level of cache, if someone goes ahead and makes a card with comparatively slower memory card slots.
dathinab 3y ago
The DDR5 bandwidth is ~51.2 GiB/s per "channel". With consumer desktop CPUs having 2 "channels" (~102.4GiB/s). And prosumer desktop/workstation CPUs (e.g. Threadripper) having 4 "channels" (~204.8GiB/s). While apple is claiming 819.2GiB/s. That is _max throughput is 4x/8x more_ (depending if you compare it to prosumer (fair comparison) or consumer (unfair comparison for a $7k system) hardware. Now the _max_ part is important, Apple mainly reaches this by having more bandwidth, i.e. parallelism. Mainly (oversimplified) the per "CPU memory channel" bandwidth for x86 is 64bit (for 51.2GiB/s) so the M1 Ultra is roughly comparable to having 16 "CPU memory channels" instead of 8 (prosumer) or 4 (consumer). Some applications can take advantage of this nicely and will scale potentially even to 4x/8x speed, most probably will not some might even have neglible improvements. But applications which use a lot of RAM (as much as they can get) and most of the RAM they use is "warm" (e.g. doesn't just lie around with little access but isn't super "hot", i.e. highly contented either) will profit quite a lot. On the other hand applications which use little RAM but the same small RAM region very heavily contented likely will hardly at all profit and will likely run faster on overclocked RAM on consumer systems. Luckily for apple most the typical use case they sell their pro desktop model for belong more/mostly in the first category. Additionally if you can run Linux on this system they might become _very_ interesting for some scientific applications for some users. I mean even e.g. Zen 4 EPYC CPU only have 12 "CPU memory channels" not 16 and it's much easier to put a desktop box "somewhere" then a server unit. Side note: I say "CPU channel" in quotes because while it tends to be the marketing term things are more complicated in practice, e.g. in general DDR5 is splitting the 64bit channels into 32bit sub-channels, and just listing the channel width and throughput is still not painting the whole picture at all, e.g. the latency also matters for some applications (hence why OC can make sense) etc. EDIT: Correction: The Threadripper PRO models have 8 "CPU memory channels" so it's just 2x on a fair comparison.