3 ms·
No, it doesn't. Where are the flops/tops?
by lostmsu 2y ago
No, it doesn't. Where are the flops/tops?
- YetAnotherNick 2y agoIt has around 1/4th of flops of H100[1], but it could still get 60% for inference if it has higher than half of memory bandwidth. [1]: https://www1.se.cuhk.edu.hk/~seem5730/l22/06%20Atlas%20AI%20Computing%20Platform_ALEX.pdf https://www1.se.cuhk.edu.hk/~seem5730/l22/06%20Atlas%20AI%20...
- lostmsu 2y agoBandwidth bottleneck applies only to non-batched inference on single accelerator setups.
- YetAnotherNick 2y agoEven for batched it applies if the batch size couldn't be super high due to memory size.
- lostmsu 2y agoIf you're memory bandwidth bound, you're using the chips wrong and may better be served by stacking DDR5.