4 ms·

Interesting that Samsung still pursues PIM. IIRC they had a paper in ISCA21 or 22 where they showed HBM2 module with PIM, which back then impressed me quite a l

Interesting that Samsung still pursues PIM. IIRC they had a paper in ISCA21 or 22 where they showed HBM2 module with PIM, which back then impressed me quite a lot. That being said, I am not sure what’s the killer application for this technology, and without such application adoption is unlikely.

xyzzy123 28d ago
As I understand it, the killer app is llms. You could run MACs directly in RAM, offloading a lot of work from CPU and cutting down on insane (external) memory bandwidth required. Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits.
- embedding-shape 28d ago
  > Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits. Isn't this how it works today already? Granted you wanted to run it on RAM rather than VRAM.
  - petu 28d ago
    Yes, but running out of RAM is impractical due to low memory bandwidth. According to the article/Samsung RAM dies inside can support way higher bandwidth than they expose, they're limited by external interface / bus width: > Together, they can utilize the chip’s internal bandwidth across all 16 banks, which comes out to 614 GB/s. For comparison, regular DRAM accesses can hit two banks in parallel and max out at 76.8 GB/s. And that's just for single 64-bit IC. So way faster and more power efficient.
    - dannyw 28d ago
      You can scale with more memory channels. Workstation/server platforms go up to 12 or 16 channels if I remember correctly. Consumer platforms have been stuck at dual channel for decades; most of it I attribute to intentional product segmentation. I'm hoping that LLMs might change eventually for an upcoming consumer platforms; going to 4 channel would be really nice.
      - zerd 27d ago
        How do you scale past 16 channels though? 16 channels give you around 614 GB/s, while PIM can do that per chip, so it can achieve 58TB/s.
- amelius 28d ago
  You: "AI, please write me $COOL_APP." AI: "Sorry, all the hardware is made for running AI."
  - eru 28d ago
    We can run Doom on everything. Surely we can run some interesting apps on hardware that's originally made for AI. (One big moment for AI was when people figured out how to run it on hardware originally meant for Doom's successors.)
- WithinReason 28d ago
  [dead]
- reliabilityguy 28d ago
  > You could run MACs directly in RAM Sure, MACs are nice. However, unless there other, PIM-specific/optimal, algorithms, regular matrix multiplication algorithms like tiling-based won’t work here I think — how would the tile be shared? By doing read/write all the time?
  - whatshisface 28d ago
    Attention calculations aren't shared across more than one vector during next token prediction (thinking and writing) which this sounds almost perfect for. Per attention layer, for deepseek at 1M context, you want to broadcast a single 1KB vector to 4GB of dot products, and map reduce a 1KB vector back.
    - reliabilityguy 28d ago
      How exactly the map-reduce will happen though? Won’t you need to do it host-side, or make a lot of reads and writes? Also, doesn’t it mean that you forgo batching?
      - deleted 28d ago
        [deleted]
      - ACCount37 28d ago
        Map-reduce is implemented as a rolling calc, see: online softmax in FlashAttention kernels.
        reliabilityguy 27d ago
        Rollie calculation, like the online softmax in FA, implies a centralized computing unit that does the compute and stores the intermediate results in its registers. With PIM you have no centralized compute unit, you have a bunch of memory, and a bunch of MACs all over the place. How would you do map-reduce across multiple DIMMs w/o extra reads/writes? PIM implies some sort of distributed compute, which can work for some cases, but I am not sure LLMs are one of them.
netfortius 28d ago
> That being said, I am not sure what's the killer application for this technology... Build it, and they will come ;)
- deleted 28d ago
  [deleted]
imtringued 28d ago
The idea is this: You have an eight socket server with 96 memory slots, you add 96x PIM memories into the server (optimistic), load all the LLM parameters or KV cache in RAM and exclusively let it perform GEMV and let it rip. 614 GB/s x 96 = 58,944 GB/s. Alternatively, the memory is used for embedded inference tasks. You can now upgrade from the limited single or two digit MB SRAM accelerators to reasonably fast single digit gigabyte models. Without MoE you could reach 100 tokens per second with an 8B fp8 model on a single channel. With MoE you might break 500 tokens per second.
- hoppp 28d ago
  And now the 96 memory slots need individual cooling.
  - throwaway173738 28d ago
    Which might be easier since the surface is larger
- reliabilityguy 27d ago
  > load all the LLM parameters or KV cache in RAM and exclusively let it perform GEMV and let it rip. Won’t you have a bunch of extra reads/writes via the CPU because these DIMMs won’t be able to compute matrix multiplications?

xyzzy123 28d ago

As I understand it, the killer app is llms. You could run MACs directly in RAM, offloading a lot of work from CPU and cutting down on insane (external) memory bandwidth required. Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits.

embedding-shape 28d ago
> Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits. Isn't this how it works today already? Granted you wanted to run it on RAM rather than VRAM.
- petu 28d ago
  Yes, but running out of RAM is impractical due to low memory bandwidth. According to the article/Samsung RAM dies inside can support way higher bandwidth than they expose, they're limited by external interface / bus width: > Together, they can utilize the chip’s internal bandwidth across all 16 banks, which comes out to 614 GB/s. For comparison, regular DRAM accesses can hit two banks in parallel and max out at 76.8 GB/s. And that's just for single 64-bit IC. So way faster and more power efficient.
  - dannyw 28d ago
    You can scale with more memory channels. Workstation/server platforms go up to 12 or 16 channels if I remember correctly. Consumer platforms have been stuck at dual channel for decades; most of it I attribute to intentional product segmentation. I'm hoping that LLMs might change eventually for an upcoming consumer platforms; going to 4 channel would be really nice.
    - zerd 27d ago
      How do you scale past 16 channels though? 16 channels give you around 614 GB/s, while PIM can do that per chip, so it can achieve 58TB/s.
amelius 28d ago
You: "AI, please write me $COOL_APP." AI: "Sorry, all the hardware is made for running AI."
- eru 28d ago
  We can run Doom on everything. Surely we can run some interesting apps on hardware that's originally made for AI. (One big moment for AI was when people figured out how to run it on hardware originally meant for Doom's successors.)
WithinReason 28d ago
[dead]
reliabilityguy 28d ago
> You could run MACs directly in RAM Sure, MACs are nice. However, unless there other, PIM-specific/optimal, algorithms, regular matrix multiplication algorithms like tiling-based won’t work here I think — how would the tile be shared? By doing read/write all the time?
- whatshisface 28d ago
  Attention calculations aren't shared across more than one vector during next token prediction (thinking and writing) which this sounds almost perfect for. Per attention layer, for deepseek at 1M context, you want to broadcast a single 1KB vector to 4GB of dot products, and map reduce a 1KB vector back.
  - reliabilityguy 28d ago
    How exactly the map-reduce will happen though? Won’t you need to do it host-side, or make a lot of reads and writes? Also, doesn’t it mean that you forgo batching?
    - deleted 28d ago
      [deleted]
    - ACCount37 28d ago
      Map-reduce is implemented as a rolling calc, see: online softmax in FlashAttention kernels.
      - reliabilityguy 27d ago
        Rollie calculation, like the online softmax in FA, implies a centralized computing unit that does the compute and stores the intermediate results in its registers. With PIM you have no centralized compute unit, you have a bunch of memory, and a bunch of MACs all over the place. How would you do map-reduce across multiple DIMMs w/o extra reads/writes? PIM implies some sort of distributed compute, which can work for some cases, but I am not sure LLMs are one of them.

netfortius 28d ago

> That being said, I am not sure what's the killer application for this technology... Build it, and they will come ;)

deleted 28d ago
[deleted]

imtringued 28d ago

The idea is this: You have an eight socket server with 96 memory slots, you add 96x PIM memories into the server (optimistic), load all the LLM parameters or KV cache in RAM and exclusively let it perform GEMV and let it rip. 614 GB/s x 96 = 58,944 GB/s. Alternatively, the memory is used for embedded inference tasks. You can now upgrade from the limited single or two digit MB SRAM accelerators to reasonably fast single digit gigabyte models. Without MoE you could reach 100 tokens per second with an 8B fp8 model on a single channel. With MoE you might break 500 tokens per second.

hoppp 28d ago
And now the 96 memory slots need individual cooling.
- throwaway173738 28d ago
  Which might be easier since the surface is larger
reliabilityguy 27d ago
> load all the LLM parameters or KV cache in RAM and exclusively let it perform GEMV and let it rip. Won’t you have a bunch of extra reads/writes via the CPU because these DIMMs won’t be able to compute matrix multiplications?