37 ms·

Does this mean that it should be possible to load up a system with ~10 (seems to me at least the number of active experts) SSDs to get 40 tok/s even on truly gi

by spwa4 8mo ago

Does this mean that it should be possible to load up a system with ~10 (seems to me at least the number of active experts) SSDs to get 40 tok/s even on truly gigantic models?

zozbot234 8mo ago
SSD bandwidth will ultimately be limited by the amount of PCIe lanes you have available (for something other than the Apple Silicon internal storage). So the approach has inherent limitations. You can of course scale out to multiple systems to get more throughput. You can use this approach with Intel Optane, which is wearout-resistant unlike NAND and can thus substitute for RAM. Last I checked, it was available quite cheap on the secondary market, ~$1/GB as opposed to ~$15/GB or more for DRAM. (Of course that's nowhere near as cheap as NAND, which is around ~$0.1/GB but quite wearout-prone with heavy writes.)
- spwa4 8mo ago
  Yeah, PCIe is the bottleneck. The point being that whether the data originates from RAM or from NVME or Optane, you cannot get data to the GPU faster with RAM than with SSDs. Meanwhile PCIe switches exist. So why not build: 1 CPU + memory + ... N PCIe switch with each 1 low-memory GPU + 6 NVME drives (in theory 5 can saturate the GPU) Each of those should only bother the CPU when they have some tokens produced and have plenty of PCIe lanes to get at their data. Such a setup should be able to get a 6 to 8 times speedup from the solution detailed here, and a model compute increase should make relatively little difference in performance.
kelnos 8mo ago
I'm not sure how you're going to get 10 SSDs to fit inside a laptop. And if you're not going to use a laptop, then you might as well get an expensive machine with plenty of RAM, no? Even with RAM prices as crazy as they are these days, that's probably still cheaper than a machine with enough PCIe bandwidth/lanes/ports for 10 SSDs, not to mention the cost of the SSDs themselves. Though I guess this could be an interesting area of research, regardless of the cost.

by spwa4 8mo ago

Does this mean that it should be possible to load up a system with ~10 (seems to me at least the number of active experts) SSDs to get 40 tok/s even on truly gigantic models?

zozbot234 8mo ago

SSD bandwidth will ultimately be limited by the amount of PCIe lanes you have available (for something other than the Apple Silicon internal storage). So the approach has inherent limitations. You can of course scale out to multiple systems to get more throughput. You can use this approach with Intel Optane, which is wearout-resistant unlike NAND and can thus substitute for RAM. Last I checked, it was available quite cheap on the secondary market, ~$1/GB as opposed to ~$15/GB or more for DRAM. (Of course that's nowhere near as cheap as NAND, which is around ~$0.1/GB but quite wearout-prone with heavy writes.)

spwa4 8mo ago
Yeah, PCIe is the bottleneck. The point being that whether the data originates from RAM or from NVME or Optane, you cannot get data to the GPU faster with RAM than with SSDs. Meanwhile PCIe switches exist. So why not build: 1 CPU + memory + ... N PCIe switch with each 1 low-memory GPU + 6 NVME drives (in theory 5 can saturate the GPU) Each of those should only bother the CPU when they have some tokens produced and have plenty of PCIe lanes to get at their data. Such a setup should be able to get a 6 to 8 times speedup from the solution detailed here, and a model compute increase should make relatively little difference in performance.

kelnos 8mo ago

I'm not sure how you're going to get 10 SSDs to fit inside a laptop. And if you're not going to use a laptop, then you might as well get an expensive machine with plenty of RAM, no? Even with RAM prices as crazy as they are these days, that's probably still cheaper than a machine with enough PCIe bandwidth/lanes/ports for 10 SSDs, not to mention the cost of the SSDs themselves. Though I guess this could be an interesting area of research, regardless of the cost.