3 ms·

Is I/O starvation the bottleneck with GPUs? I didn't think it was.

• hmottestad 2y ago
  Memory bandwidth is the bottleneck for LLM inference. That's my understanding at least.
  • moffkalast 2y ago
    That's correct, but also compute to some degree. The larger the model the more of a bottleneck memory becomes. There are some older HBM cards with very high bandwidth like the Radeon Pro VII which has 1TB/s of bandwidth like the RTX 3090 and 4090, but is notably slower at inference for smaller models since has less compute in comparison. At least I think that was the consensus of some benchmarks people ran.
  • littlestymaar 2y ago
    Isn't it only the case when inference isn't batched?
    • Tostino 2y ago
      Even in a local setting, batched inference is useful to be able to run more "complex" workflows (with multiple, parallel LLM calls for a single interaction). There is very little reason to optimize for just single stream inference at the expense of your batch inference performance.
• alwayslikethis 2y ago
  For inference, it often is. Though for most consumer parts the bigger concern is not having enough VRAM rather than the VRAM not being fast enough. Copying from system RAM to VRAM is far slower.
• mmoskal 2y ago
  With a typical transformer and a GPU the batch size that saturates the compute is at least hundreds. Otherwise (including typical size of 1 for local inference) you're memory bound.
• corysama 2y ago
  GPUs have much more memory bandwidth than CPUs. Meanwhile, the ALU:bandwidth ratio of both GPUs and CPUs has been growing exponentially since the 90s at least. So, the FLOPs per byte required to not be starved on memory is really large at this point. We’re at a point that optimization is 90% about SRAM utilization and you worry about the math maybe at the last step.
• deleted 2y ago
  [deleted]