Almost all of this seems to be the same or similar to distributing computation over the internet. At least the main concepts like "sending work to your data". A

Almost all of this seems to be the same or similar to distributing computation over the internet. At least the main concepts like "sending work to your data". And I think they are talking about CPUs, but that main idea also could pertain to GPUs. I wonder if there is a level inside of modern CPU cores where cache is shared by different parallel pipelines or sub-caches for each. It's sort of fractal. Moving the data around is the bottleneck, so the most efficient way is to arrange the data so that data movement is minimized, and that applies at different zoom levels. I suspect that there is a lot of research in memory-optimized computing that are possibly different enough from typical approaches to be considered new paradigms. I wonder if this could intersect with something like Mixture of Experts in LLMs. I don't actually know how that works, but maybe it could allow for grouping the related expert weight data closer together to speed things up. Also, think about the complexity of real neurons when compared to something like those in an MLP. My impression is that real neurons pack in (locally) much more information and compute per unit. I wonder if you could use some type of model to predict what data you will need for text generation (aside from the KV cache of course), and then have a fast cache for each task. Is there a large open LLM or LMM trained using something like https://github.com/lucidrains/st-moe-pytorch https://github.com/lucidrains/st-moe-pytorch ?