It is not at all just quantities of memory (or speed of computation) It's in large part a problem of bandwidth, too. Mostly really. HBM memory can do up to 3TB

It is not at all just quantities of memory (or speed of computation) It's in large part a problem of bandwidth, too. Mostly really. HBM memory can do up to 3TB/s vs DDR5 like 250GB/S. The latter is just too slow to process 2.5B parameter models, it simply can't move the values back and forth fast enough. It would drag to a crawl. Much smaller dense models at that speed on the NVIDIA Spark can't do more than 15tok/sec. Real serving systems for these models involve large numbers of parallel GPUs with massive memory bandwidth, hooked up via NVlink. It will take a long time for that level of tech to get down to consumer level. (An ideal computing architecture built for LLMs would in fact offer some way of colocating computation with memory. If you can put matmul etc right in the DRAM and avoid going back and forth over the bus...)