4 ms·
As I understand it, the killer app is llms. You could run MACs directly in RAM, offloading a lot of work from CPU and cutting down on insane (external) memory b
by xyzzy123 1mo ago
As I understand it, the killer app is llms. You could run MACs directly in RAM, offloading a lot of work from CPU and cutting down on insane (external) memory bandwidth required.
Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits.
- embedding-shape 1mo ago> Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits. Isn't this how it works today already? Granted you wanted to run it on RAM rather than VRAM.
- petu 1mo agoYes, but running out of RAM is impractical due to low memory bandwidth. According to the article/Samsung RAM dies inside can support way higher bandwidth than they expose, they're limited by external interface / bus width: > Together, they can utilize the chip’s internal bandwidth across all 16 banks, which comes out to 614 GB/s. For comparison, regular DRAM accesses can hit two banks in parallel and max out at 76.8 GB/s. And that's just for single 64-bit IC. So way faster and more power efficient.
- dannyw 1mo agoYou can scale with more memory channels. Workstation/server platforms go up to 12 or 16 channels if I remember correctly. Consumer platforms have been stuck at dual channel for decades; most of it I attribute to intentional product segmentation. I'm hoping that LLMs might change eventually for an upcoming consumer platforms; going to 4 channel would be really nice.
- zerd 1mo agoHow do you scale past 16 channels though? 16 channels give you around 614 GB/s, while PIM can do that per chip, so it can achieve 58TB/s.
- amelius 1mo agoYou: "AI, please write me $COOL_APP." AI: "Sorry, all the hardware is made for running AI."
- eru 1mo agoWe can run Doom on everything. Surely we can run some interesting apps on hardware that's originally made for AI. (One big moment for AI was when people figured out how to run it on hardware originally meant for Doom's successors.)
- WithinReason 1mo ago[dead]
- reliabilityguy 1mo ago> You could run MACs directly in RAM Sure, MACs are nice. However, unless there other, PIM-specific/optimal, algorithms, regular matrix multiplication algorithms like tiling-based won’t work here I think — how would the tile be shared? By doing read/write all the time?
- whatshisface 1mo agoAttention calculations aren't shared across more than one vector during next token prediction (thinking and writing) which this sounds almost perfect for. Per attention layer, for deepseek at 1M context, you want to broadcast a single 1KB vector to 4GB of dot products, and map reduce a 1KB vector back.
- reliabilityguy 1mo agoHow exactly the map-reduce will happen though? Won’t you need to do it host-side, or make a lot of reads and writes? Also, doesn’t it mean that you forgo batching?
- deleted 1mo ago[deleted]
- ACCount37 1mo agoMap-reduce is implemented as a rolling calc, see: online softmax in FlashAttention kernels.
- reliabilityguy 1mo agoRollie calculation, like the online softmax in FA, implies a centralized computing unit that does the compute and stores the intermediate results in its registers. With PIM you have no centralized compute unit, you have a bunch of memory, and a bunch of MACs all over the place. How would you do map-reduce across multiple DIMMs w/o extra reads/writes? PIM implies some sort of distributed compute, which can work for some cases, but I am not sure LLMs are one of them.