3 ms·
Bandwidth bottleneck applies only to non-batched inference on single accelerator setups.
by lostmsu 2y ago
Bandwidth bottleneck applies only to non-batched inference on single accelerator setups.
- YetAnotherNick 2y agoEven for batched it applies if the batch size couldn't be super high due to memory size.
- lostmsu 2y agoIf you're memory bandwidth bound, you're using the chips wrong and may better be served by stacking DDR5.