3 ms·
This is on llama 3.1 405B. Inferencing is memory bandwidth bound. Add more GPUs on a batch size 1 inference problem and watch it run no faster than the memory
by ryao 2y ago
This is on llama 3.1 405B.
Inferencing is memory bandwidth bound. Add more GPUs on a batch size 1 inference problem and watch it run no faster than the memory bandwidth of a single GPU. It does not scale across the number of GPUs. If it could, you would see clusters of Nvidia hardware outperforming Cerebras’ hardware. That is currently a fantasy.
- YetAnotherNick 2y agoThis two sources[1][2] shows 1500-2500 token/per second on 8*H100. [1]: https://lmsys.org/blog/2024-07-25-sglang-llama3/?ref=blog.runpod.io https://lmsys.org/blog/2024-07-25-sglang-llama3/?ref=blog.ru... [2]: https://www.snowflake.com/engineering-blog/optimize-llms-with-llama-snowflake-ai-stack/ https://www.snowflake.com/engineering-blog/optimize-llms-wit...