4 ms·

Why would you want/need to? The advantage of HBM over regular non-stacked DRAM is memory bandwidth, which also requires a super-wide memory bus - 2048 bits wid

Why would you want/need to? The advantage of HBM over regular non-stacked DRAM is memory bandwidth, which also requires a super-wide memory bus - 2048 bits wide for HBM4. Compare that to the 128 bit wide bus of a modern CPU. So to take advantage of it on the desktop, or anywhere else, you need that 2048 bit wide bus, and a processor capable of consuming 2-3 TB of data per second! These are not normal requirements, other than for a GPU.

fooker 6d ago
SIMD (and especially the modern matrix extensions) can use as much bandwidth you can throw at it.
- b112 6d ago
  Without further clarification, that statement seems impossible. "As much" being unbounded and all. You should expand what you mean.
  - fooker 6d ago
    This will blow your mind, but it actually is pretty close to being unbounded. :) Consider the 'MMA N matrices' primitive modern CPUs are starting to support. For the current generation of CPUs, N is a constant like 16 or 32, but there's nothing preventing it from being 1024 or larger if we have more memory bandwidth. All this with a single instruction.
    - articulatepang 6d ago
      Surely something prevents it being 1 quadrillion bits per instruction? Since that’s well within “unbounded”.
      - pixl97 5d ago
        Most likely the chip running at the core temperature of the sun. We'll have to figure out how to read and right to the surface of a black hole to get speeds that high.
    - imtringued 5d ago
      Nah his point is a bit simpler. Vector units can process a limited amount of data per time unit. Memory can load a limited amount of data per time unit. If you have infinite memory bandwidth you just move the bottleneck back to compute so both have to grow simultaneously in lockstep. What you should have said is that CPUs have so much compute headroom for matrix vector multiplication that simply adding more memory bandwidth would make them faster so every improvement in memory bandwidth is welcome.
      - fooker 5d ago
        Agreed. The "move the bottleneck back to compute" bit is changing rapidly though. The first time a major hardware company ships a PIM chip, you can push for a few orders of magnitude more data through without being compute bound.
XorNot 6d ago
Right but if HBM memory is all that people want to produce, then building a CPU which can use it use it would be useful on it's own merits. But in reality we also already have unified memory architecture systems, integrated graphics etc.
- p1esk 6d ago
  People want to produce hbm because it’s more expensive and more profitable than regular memory.
  - XorNot 6d ago
    There is a world where scale and experience means it's about the same though, is the thing. And memory is already expensive. It's downright hard to even get it though - you frequently would prefer not what's cheapest, but whatever is in largest scale production.
    - Dylan16807 6d ago
      Scale and experience almost entirely share between HBM and normal memory. And they're both in large-enough scale production to not have a big difference on availability; if you're willing to pay HBM prices you should find even more sellers of DDR. The only way I see HBM becoming competitive for consumer CPUs is if they solve the yield issues. Or if AI crashes so hard that people are putting those GPUs on fire sale and salvaging mass quantities of HBM off of them.
      - zeristor 6d ago
        I was thinking this, but I doubt that the HBM is so easy to repurpose. I’m assuming that it’s mounted in the same unit as the GPU, not in a handy-dandy socket. It could be cracked open and extracted no doubt but if it’s glued in that’s going to be nigh on impossible to extract. I had been pinning my hopes on HBM coming onto the second hand markets after there three years or so of use, perhaps I’m wrong.
Dylan16807 6d ago
> Compare that to the 128 bit wide bus of a modern CPU. Or 256-512 bits on medium to high end consumer CPUs if you're apple. At least DDR6 is probably widening things 50%.
bobmcnamara 6d ago
Intel and IBM have already done this almost this with their wide eDRAM caches. The advantage in going wide is transferring cache lines rapidly, not the CPU bus interface.

fooker 6d ago

SIMD (and especially the modern matrix extensions) can use as much bandwidth you can throw at it.

b112 6d ago
Without further clarification, that statement seems impossible. "As much" being unbounded and all. You should expand what you mean.
- fooker 6d ago
  This will blow your mind, but it actually is pretty close to being unbounded. :) Consider the 'MMA N matrices' primitive modern CPUs are starting to support. For the current generation of CPUs, N is a constant like 16 or 32, but there's nothing preventing it from being 1024 or larger if we have more memory bandwidth. All this with a single instruction.
  - articulatepang 6d ago
    Surely something prevents it being 1 quadrillion bits per instruction? Since that’s well within “unbounded”.
    - pixl97 5d ago
      Most likely the chip running at the core temperature of the sun. We'll have to figure out how to read and right to the surface of a black hole to get speeds that high.
  - imtringued 5d ago
    Nah his point is a bit simpler. Vector units can process a limited amount of data per time unit. Memory can load a limited amount of data per time unit. If you have infinite memory bandwidth you just move the bottleneck back to compute so both have to grow simultaneously in lockstep. What you should have said is that CPUs have so much compute headroom for matrix vector multiplication that simply adding more memory bandwidth would make them faster so every improvement in memory bandwidth is welcome.
    - fooker 5d ago
      Agreed. The "move the bottleneck back to compute" bit is changing rapidly though. The first time a major hardware company ships a PIM chip, you can push for a few orders of magnitude more data through without being compute bound.

XorNot 6d ago

Right but if HBM memory is all that people want to produce, then building a CPU which can use it use it would be useful on it's own merits. But in reality we also already have unified memory architecture systems, integrated graphics etc.

p1esk 6d ago
People want to produce hbm because it’s more expensive and more profitable than regular memory.
- XorNot 6d ago
  There is a world where scale and experience means it's about the same though, is the thing. And memory is already expensive. It's downright hard to even get it though - you frequently would prefer not what's cheapest, but whatever is in largest scale production.
  - Dylan16807 6d ago
    Scale and experience almost entirely share between HBM and normal memory. And they're both in large-enough scale production to not have a big difference on availability; if you're willing to pay HBM prices you should find even more sellers of DDR. The only way I see HBM becoming competitive for consumer CPUs is if they solve the yield issues. Or if AI crashes so hard that people are putting those GPUs on fire sale and salvaging mass quantities of HBM off of them.
    - zeristor 6d ago
      I was thinking this, but I doubt that the HBM is so easy to repurpose. I’m assuming that it’s mounted in the same unit as the GPU, not in a handy-dandy socket. It could be cracked open and extracted no doubt but if it’s glued in that’s going to be nigh on impossible to extract. I had been pinning my hopes on HBM coming onto the second hand markets after there three years or so of use, perhaps I’m wrong.

Dylan16807 6d ago

> Compare that to the 128 bit wide bus of a modern CPU. Or 256-512 bits on medium to high end consumer CPUs if you're apple. At least DDR6 is probably widening things 50%.

bobmcnamara 6d ago

Intel and IBM have already done this almost this with their wide eDRAM caches. The advantage in going wide is transferring cache lines rapidly, not the CPU bus interface.