4 ms·
That comparison doesn't make sense. If you want 192GB VRAM, then that is the size of 8 4090s, so to make it competitive, the bandwidth also needs to be 8 times
by andersa 2y ago
That comparison doesn't make sense. If you want 192GB VRAM, then that is the size of 8 4090s, so to make it competitive, the bandwidth also needs to be 8 times higher. Or 8TB/s in this case.
- behnamoh 2y agoThis is so wrong I don't even need to correct it.
- saagarjha 2y agoYou don't need to post like that, either. And yet here we are.
- klohto 2y agoWhat are you talking about? Why would you multiply the bandwidth? 8 4090s is still 1000GB/s. While the M2 Ultra is 800GB/s with a top of 192GB VRAM. Metal can access ~155GB, so you need a bit more, but your comparison makes absolutely zero sense.
- boroboro4 2y agoThere are different ways to run LLMs on multiple GPUs, one of them (called tensor parallelism) in low batch scenarios would be multiplying bandwidth between different GPUs. So no, 8 4090s is not 1000 GB/s.
- klohto 2y agolet me know how is the PCIe bandwidth treating you
- boroboro4 2y agoSince we’re talking about small batch sizes PCIe bandwidth isn’t as important - intermediate hidden state is magnitude smaller than weights.
- behnamoh 2y agoyou've heard something and are regurgitating it without fully understanding it.
- boroboro4 2y agoI’m developing inference engine, so I actually do understand how it works. As well as other types of parallelism and how exactly they do different trade offs
- andersa 2y agoWhen you have 8 GPUs, you can use more than 1 at a time.