3 ms·
It has around 1/4th of flops of H100[1], but it could still get 60% for inference if it has higher than half of memory bandwidth. [1]: https://www1.se.cuhk.edu
by YetAnotherNick 2y ago
It has around 1/4th of flops of H100[1], but it could still get 60% for inference if it has higher than half of memory bandwidth.
[1]: https://www1.se.cuhk.edu.hk/~seem5730/l22/06%20Atlas%20AI%20Computing%20Platform_ALEX.pdf https://www1.se.cuhk.edu.hk/~seem5730/l22/06%20Atlas%20AI%20...
- lostmsu 2y agoBandwidth bottleneck applies only to non-batched inference on single accelerator setups.
- YetAnotherNick 2y agoEven for batched it applies if the batch size couldn't be super high due to memory size.
- lostmsu 2y agoIf you're memory bandwidth bound, you're using the chips wrong and may better be served by stacking DDR5.