2 ms·

taps the sign Unified Memory Is A Marketing Gimmeck. Industrial-Scale Inference Servers Do Not Use It.

zozbot234 7mo ago
Industrial Scale Inference is moving towards LPDDR memory (alongside HBM), which is essentially what "Unified Memory" is.
- bigyabai 7mo ago
  LPDDR is LPDDR. There's nothing "unified" about it architecturally.
- 0x457 7mo ago
  > which is essentially what "Unified Memory" is. Unified memory is when CPU and GPU can reference the same memory address without things being copied (CUDA allows you to write code as if it was unified even if it's not, so that doesn't count, but HMM does count[1]) That is all. What technology is underneath is hardware detail. Unified memory on macs lets you put something into a memory, then do some computation on it with CPU, ANE, ANA, Metal Shaders. All without copying anything. DGX Spark also has unified memory. [1]: https://docs.nvidia.com/cuda/cuda-programming-guide/02-basics/understanding-memory.html https://docs.nvidia.com/cuda/cuda-programming-guide/02-basic...
rcxdude 7mo ago
Unified Memory is mainly how consumer hardware has enough RAM accessible by the GPU to run larger models, because otherwise the market segmentation jacks up the price substantially.
- bigyabai 7mo ago
  UMA removes the PCIe bottleneck and replaces it with a memory controller + bandwidth bottleneck. For most high-performance GPUs, that would be a direct downgrade.
  - deleted 7mo ago
    [deleted]
wren6991 7mo ago
On M5 Pro/Max the memory is actually just attached straight to the GPU die. CPU accesses memory through the die-to-die bridge. I don't see the difference between that and a pure GPU from a memory connectivity point of view. Wrt inference servers: sure, it's not cost-effective to have such a huge CPU die and a bunch of media accelerators on the GPU die if you just care about raw compute for inference and training. Apple SoCs are not tuned for that market, nor do they sell into it. I'm not building a datacentre, I'm trying to run inference on my home hardware that I also want to use for other things.
- bigyabai 7mo ago
  If you're going to do unified memory, that's the way to do it, in addition to using higher-bandwidth RAM and padding out your GPGPU hardware. Nvidia realized this almost a decade ago, and Apple is being dragged through the mud to learn the exact same $4 trillion lesson.

Industrial Scale Inference is moving towards LPDDR memory (alongside HBM), which is essentially what "Unified Memory" is.

bigyabai 7mo ago
LPDDR is LPDDR. There's nothing "unified" about it architecturally.
0x457 7mo ago
> which is essentially what "Unified Memory" is. Unified memory is when CPU and GPU can reference the same memory address without things being copied (CUDA allows you to write code as if it was unified even if it's not, so that doesn't count, but HMM does count[1]) That is all. What technology is underneath is hardware detail. Unified memory on macs lets you put something into a memory, then do some computation on it with CPU, ANE, ANA, Metal Shaders. All without copying anything. DGX Spark also has unified memory. [1]: https://docs.nvidia.com/cuda/cuda-programming-guide/02-basics/understanding-memory.html https://docs.nvidia.com/cuda/cuda-programming-guide/02-basic...

rcxdude 7mo ago

Unified Memory is mainly how consumer hardware has enough RAM accessible by the GPU to run larger models, because otherwise the market segmentation jacks up the price substantially.

bigyabai 7mo ago
UMA removes the PCIe bottleneck and replaces it with a memory controller + bandwidth bottleneck. For most high-performance GPUs, that would be a direct downgrade.
- deleted 7mo ago
  [deleted]

wren6991 7mo ago

On M5 Pro/Max the memory is actually just attached straight to the GPU die. CPU accesses memory through the die-to-die bridge. I don't see the difference between that and a pure GPU from a memory connectivity point of view. Wrt inference servers: sure, it's not cost-effective to have such a huge CPU die and a bunch of media accelerators on the GPU die if you just care about raw compute for inference and training. Apple SoCs are not tuned for that market, nor do they sell into it. I'm not building a datacentre, I'm trying to run inference on my home hardware that I also want to use for other things.

bigyabai 7mo ago
If you're going to do unified memory, that's the way to do it, in addition to using higher-bandwidth RAM and padding out your GPGPU hardware. Nvidia realized this almost a decade ago, and Apple is being dragged through the mud to learn the exact same $4 trillion lesson.