PCIe 5 NVMe drives can do 11+ GBps so at least 2x your numbers. We seem to be talking past each other because the point of the change is to run inference on a c

PCIe 5 NVMe drives can do 11+ GBps so at least 2x your numbers. We seem to be talking past each other because the point of the change is to run inference on a cpu for an LLM with a larger weight size than the host RAM can fit. It looks to me that if I was planning on building a new machine capable of LLM inference it’s going to be possible using commodity gamer components and if lazy weights is viable, then such a machine with multiple PCIe 5 nvme drives in a raid 0 can potentially almost reach memory bandwidth. On my list of to investigate next is in regards to inference with GPUs, could somehow multiple smaller GPUs be used with a technique similar to the OP post.