3 ms·
You can improve that with speculative preload. I'm sure models could be designed and tuned around efficient SSD offloading to keep throughput pretty high.
by CuriouslyC 3mo ago
You can improve that with speculative preload. I'm sure models could be designed and tuned around efficient SSD offloading to keep throughput pretty high.
- rsalus 3mo agosurely the supply of unified memory will rise to meet demand before this is needed
- searealist 3mo agoIt would apply equally to GPU or RAM inference as those are also bandwidth constrained on decode, so people already try to optimize for it.