4 ms·

What they seem to want is fast-read, slow-write memory. "Primary applications include model weights in ML inference, code pages, hot instruction paths, and rela

What they seem to want is fast-read, slow-write memory. "Primary applications include model weights in ML inference, code pages, hot instruction paths, and relatively static data pages". Is there device physics for cheaper, smaller fast-read slow write memory cells for that? For "hot instruction paths", caching is already the answer. Not sure about locality of reference for model weights. Do LLMs blow the cache?

toast0 1y ago
Probably not what they want, but NOR flash is generally directly addressable, it's commonly used to replace mask roms.
- bobmcnamara 1y ago
  NOR is usually limited to <30MHz, but if you always want to fetch an entire cacheline, and design the read port, you can fetch the entire cacheline at once so that's pretty neat. I don't know if anyone has applied this to neutral networks.
- kimixa 1y ago
  And does so by being larger, with the only real difference being more area (and gates, and often then critical path length and thus speed) being spent on getting the signaling to the level where it can do word instead of page addressing. The actual flash cells themselves are functionally the same. There's no fundamental difference in gate technology between the two, so a flash that is addressable to a finder granularity will always be larger than the coarser equivalent. That's the trade off.
bobmcnamara 1y ago
> Do LLMs blow the cache? Sometimes very yes? If you've got 1GB of weights, those are coming through the caches on their way to execution unit somehow. Many caches are smart enough to recognize these accesses as a strided, streaming, heavily prefetchable, evictable read, and optimize for that. Many models are now quantized too to reduce the overall the overall memory bandwidth needed for execution, which also helps with caching.
photochemsyn 1y ago
Yes, this from the paper: > "The key insight motivating LtRAM is that long data lifetimes and read heavy access patterns allow optimizations that are unsuitable for general purpose memories. Primary applications include model weights in ML inference, code pages, hot instruction paths, and relatively static data pages—workloads that can tolerate higher write costs in exchange for lower read energy and improved cost per bit. This specialization addresses fundamental mismatches in current systems where read intensive data competes for the same resources as frequently modified data." Essentially I guess they're calling for more specific hardware for LLM tasks, much like was done with all the networking equipment for dedicated packet processing with specialized SRAM/DRAM/TCAM tiers to keep latency to a minimum. While there's an obvious need for this for traffic flow across the internet, whether or not LLMs are really going to scale like that, or there's a massive AI/LLM bubble about to pop, would be the practical issue, and who knows? The tea leaves are unclear.
gary_0 1y ago
> device physics for cheaper, smaller And lower power usage. Datacenters and mobile devices will always want that.
kayson 1y ago
Cheaper / smaller? I would say not likely. There is already an enormous amount of market pressure to make SRAM and DRAM smaller. Device physics-wise, you could probably make SRAM faster by dropping the transistor threshold voltage. It would also make it harder / slower to write. The bigger downside is that it would have higher leakage power, but if it's a small portion of all the SRAM, it might be worth the tradeoff. For DRAM, there isn't as much "device" involved because the storage element isn't transistor-based. You could probably make some design tradeoff in the sense amplifier to reduce read times by trading off write times, but I doubt it would make a significant change.
- kimixa 1y ago
  But much of the latency in cache is getting the signal to and from the cell, not the actual store threshold. And I can't see much difference in that unless you can actually eliminate gates (and so make it smaller, making it physically closer on average).

toast0 1y ago

Probably not what they want, but NOR flash is generally directly addressable, it's commonly used to replace mask roms.

bobmcnamara 1y ago
NOR is usually limited to <30MHz, but if you always want to fetch an entire cacheline, and design the read port, you can fetch the entire cacheline at once so that's pretty neat. I don't know if anyone has applied this to neutral networks.
kimixa 1y ago
And does so by being larger, with the only real difference being more area (and gates, and often then critical path length and thus speed) being spent on getting the signaling to the level where it can do word instead of page addressing. The actual flash cells themselves are functionally the same. There's no fundamental difference in gate technology between the two, so a flash that is addressable to a finder granularity will always be larger than the coarser equivalent. That's the trade off.

bobmcnamara 1y ago

> Do LLMs blow the cache? Sometimes very yes? If you've got 1GB of weights, those are coming through the caches on their way to execution unit somehow. Many caches are smart enough to recognize these accesses as a strided, streaming, heavily prefetchable, evictable read, and optimize for that. Many models are now quantized too to reduce the overall the overall memory bandwidth needed for execution, which also helps with caching.

photochemsyn 1y ago

Yes, this from the paper: > "The key insight motivating LtRAM is that long data lifetimes and read heavy access patterns allow optimizations that are unsuitable for general purpose memories. Primary applications include model weights in ML inference, code pages, hot instruction paths, and relatively static data pages—workloads that can tolerate higher write costs in exchange for lower read energy and improved cost per bit. This specialization addresses fundamental mismatches in current systems where read intensive data competes for the same resources as frequently modified data." Essentially I guess they're calling for more specific hardware for LLM tasks, much like was done with all the networking equipment for dedicated packet processing with specialized SRAM/DRAM/TCAM tiers to keep latency to a minimum. While there's an obvious need for this for traffic flow across the internet, whether or not LLMs are really going to scale like that, or there's a massive AI/LLM bubble about to pop, would be the practical issue, and who knows? The tea leaves are unclear.

gary_0 1y ago

> device physics for cheaper, smaller And lower power usage. Datacenters and mobile devices will always want that.

kayson 1y ago

Cheaper / smaller? I would say not likely. There is already an enormous amount of market pressure to make SRAM and DRAM smaller. Device physics-wise, you could probably make SRAM faster by dropping the transistor threshold voltage. It would also make it harder / slower to write. The bigger downside is that it would have higher leakage power, but if it's a small portion of all the SRAM, it might be worth the tradeoff. For DRAM, there isn't as much "device" involved because the storage element isn't transistor-based. You could probably make some design tradeoff in the sense amplifier to reduce read times by trading off write times, but I doubt it would make a significant change.

kimixa 1y ago
But much of the latency in cache is getting the signal to and from the cell, not the actual store threshold. And I can't see much difference in that unless you can actually eliminate gates (and so make it smaller, making it physically closer on average).