See: https://en.m.wikipedia.org/wiki/Random-access_memory#Memory_wall https://en.m.wikipedia.org/wiki/Random-access_memory#Memory_... But llama.cpp, and llms i

See: https://en.m.wikipedia.org/wiki/Random-access_memory#Memory_wall https://en.m.wikipedia.org/wiki/Random-access_memory#Memory_... But llama.cpp, and llms in general, are very atypically memory bound, even for AI workloads. Other models/backends will typically make more use of compute and cache. For llms specifically, cloud hosts will batch requests to make better use of GPUs. But you are not far off. There is a proposition to pipe chips with very fast memory (aka no external memory at all) together: https://www.nextplatform.com/2023/07/12/microsofts-chiplet-cloud-to-bring-the-cost-of-llms-way-down/ https://www.nextplatform.com/2023/07/12/microsofts-chiplet-c...