3 ms·

I'll be the one to ask the obvious question: What does this mean for compute workloads? Specifically, LLM inference. Does it mean anything at all, or is this

• skew-aberration 2mo ago
  I doubt it makes much of a difference, and you can always manually manage what data lives in the GPU when if you 100% have to overcommit. Games have a much larger and more diverse set of objects in the VRAM, and their usage is less predictable, so manual scheduling of the memory is infeasible typically.
• zozbot234 2mo ago
  If you have model layers/experts that reside in CPU RAM, it's generally better to do that part of inference on the CPU than pay the cost of shipping them over to the GPU. LLM inference is generally bottlenecked by data-movement, not compute. There may be a limited exception for prefill or perhaps decode of very wide batches, where shuffling the data around may be justified.