4 ms·
There are different ways to run LLMs on multiple GPUs, one of them (called tensor parallelism) in low batch scenarios would be multiplying bandwidth between dif
by boroboro4 2y ago
There are different ways to run LLMs on multiple GPUs, one of them (called tensor parallelism) in low batch scenarios would be multiplying bandwidth between different GPUs. So no, 8 4090s is not 1000 GB/s.
- klohto 2y agolet me know how is the PCIe bandwidth treating you
- boroboro4 2y agoSince we’re talking about small batch sizes PCIe bandwidth isn’t as important - intermediate hidden state is magnitude smaller than weights.
- behnamoh 2y agoyou've heard something and are regurgitating it without fully understanding it.
- boroboro4 2y agoI’m developing inference engine, so I actually do understand how it works. As well as other types of parallelism and how exactly they do different trade offs