4 ms·

More benchmaxxing I see. Too bad there’s no rig with 256gb unified ram for under $1000

kennethops 6mo ago
do you know if they did this to it? https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/ https://research.google/blog/turboquant-redefining-ai-effici...
- kgeist 6mo ago
  Llama.cpp already uses an idea from it internally for the KV cache [0] So a quantized KV cache now must see less degradation [0] https://github.com/ggml-org/llama.cpp/pull/21038 https://github.com/ggml-org/llama.cpp/pull/21038
bigyabai 6mo ago
taps the sign Unified Memory Is A Marketing Gimmeck. Industrial-Scale Inference Servers Do Not Use It.
- zozbot234 6mo ago
  Industrial Scale Inference is moving towards LPDDR memory (alongside HBM), which is essentially what "Unified Memory" is.
  - bigyabai 6mo ago
    LPDDR is LPDDR. There's nothing "unified" about it architecturally.
  - 0x457 6mo ago
    > which is essentially what "Unified Memory" is. Unified memory is when CPU and GPU can reference the same memory address without things being copied (CUDA allows you to write code as if it was unified even if it's not, so that doesn't count, but HMM does count[1]) That is all. What technology is underneath is hardware detail. Unified memory on macs lets you put something into a memory, then do some computation on it with CPU, ANE, ANA, Metal Shaders. All without copying anything. DGX Spark also has unified memory. [1]: https://docs.nvidia.com/cuda/cuda-programming-guide/02-basics/understanding-memory.html https://docs.nvidia.com/cuda/cuda-programming-guide/02-basic...
- rcxdude 6mo ago
  Unified Memory is mainly how consumer hardware has enough RAM accessible by the GPU to run larger models, because otherwise the market segmentation jacks up the price substantially.
  - bigyabai 6mo ago
    UMA removes the PCIe bottleneck and replaces it with a memory controller + bandwidth bottleneck. For most high-performance GPUs, that would be a direct downgrade.
    - deleted 6mo ago
      [deleted]
- wren6991 6mo ago
  On M5 Pro/Max the memory is actually just attached straight to the GPU die. CPU accesses memory through the die-to-die bridge. I don't see the difference between that and a pure GPU from a memory connectivity point of view. Wrt inference servers: sure, it's not cost-effective to have such a huge CPU die and a bunch of media accelerators on the GPU die if you just care about raw compute for inference and training. Apple SoCs are not tuned for that market, nor do they sell into it. I'm not building a datacentre, I'm trying to run inference on my home hardware that I also want to use for other things.
  - bigyabai 6mo ago
    If you're going to do unified memory, that's the way to do it, in addition to using higher-bandwidth RAM and padding out your GPGPU hardware. Nvidia realized this almost a decade ago, and Apple is being dragged through the mud to learn the exact same $4 trillion lesson.
cpburns2009 6mo ago
Sir, this is 2026. You're not getting any 128GB of RAM for under $1k.

do you know if they did this to it? https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/ https://research.google/blog/turboquant-redefining-ai-effici...

kgeist 6mo ago
Llama.cpp already uses an idea from it internally for the KV cache [0] So a quantized KV cache now must see less degradation [0] https://github.com/ggml-org/llama.cpp/pull/21038 https://github.com/ggml-org/llama.cpp/pull/21038

bigyabai 6mo ago

taps the sign Unified Memory Is A Marketing Gimmeck. Industrial-Scale Inference Servers Do Not Use It.

zozbot234 6mo ago
Industrial Scale Inference is moving towards LPDDR memory (alongside HBM), which is essentially what "Unified Memory" is.
- bigyabai 6mo ago
  LPDDR is LPDDR. There's nothing "unified" about it architecturally.
- 0x457 6mo ago
  > which is essentially what "Unified Memory" is. Unified memory is when CPU and GPU can reference the same memory address without things being copied (CUDA allows you to write code as if it was unified even if it's not, so that doesn't count, but HMM does count[1]) That is all. What technology is underneath is hardware detail. Unified memory on macs lets you put something into a memory, then do some computation on it with CPU, ANE, ANA, Metal Shaders. All without copying anything. DGX Spark also has unified memory. [1]: https://docs.nvidia.com/cuda/cuda-programming-guide/02-basics/understanding-memory.html https://docs.nvidia.com/cuda/cuda-programming-guide/02-basic...
rcxdude 6mo ago
Unified Memory is mainly how consumer hardware has enough RAM accessible by the GPU to run larger models, because otherwise the market segmentation jacks up the price substantially.
- bigyabai 6mo ago
  UMA removes the PCIe bottleneck and replaces it with a memory controller + bandwidth bottleneck. For most high-performance GPUs, that would be a direct downgrade.
  - deleted 6mo ago
    [deleted]
wren6991 6mo ago
On M5 Pro/Max the memory is actually just attached straight to the GPU die. CPU accesses memory through the die-to-die bridge. I don't see the difference between that and a pure GPU from a memory connectivity point of view. Wrt inference servers: sure, it's not cost-effective to have such a huge CPU die and a bunch of media accelerators on the GPU die if you just care about raw compute for inference and training. Apple SoCs are not tuned for that market, nor do they sell into it. I'm not building a datacentre, I'm trying to run inference on my home hardware that I also want to use for other things.
- bigyabai 6mo ago
  If you're going to do unified memory, that's the way to do it, in addition to using higher-bandwidth RAM and padding out your GPGPU hardware. Nvidia realized this almost a decade ago, and Apple is being dragged through the mud to learn the exact same $4 trillion lesson.

cpburns2009 6mo ago

Sir, this is 2026. You're not getting any 128GB of RAM for under $1k.