Right. It's super interesting to me because some years ago I got dinner with a director of AI research at Google and he told me the LLMs at that time were super

Right. It's super interesting to me because some years ago I got dinner with a director of AI research at Google and he told me the LLMs at that time were super sparse. Not sure if something got lost in translation or stuff just changed, but it doesn't seem to be true anymore. In theory NVIDIA and others could optimize for sparse matrices, right? If the operands are that sparse I wonder if whole tiles could be trivially zeroed without ever executing a matmul at all. The problem feels more like RAM and how you efficiently encode such a sparse entity without wasting lots of memory and bandwidth transferring zeros around. You can use RLE but if you have to unpack into memory to use the hardware anyway maybe it's not a win in the end.