2 ms·
What if they put 8 identical GPUs in the package, each with 1/8 the memory? Would that be a useful configuration for a modern LLM?
by amelius 2y ago
What if they put 8 identical GPUs in the package, each with 1/8 the memory? Would that be a useful configuration for a modern LLM?
- keyboard_slap 2y agoIt could work, but would it be cost-competitive?
- rini17 2y agoAlso, cooling.
- ben_w 2y agoLast I've heard, the architecture makes that difficult. But my information may be outdated, and even if it isn't, I'm not a hardware designer and may have just misunderstood the limits I hear others discuss.
- numeri 2y agoGPU inference is always a balancing act, trying to avoid bottlenecks on memory bandwidth (loading data from the GPU's global memory/VRAM to the much smaller internal shared memory, where it can be used for calculations) and compute (once the values are loaded). Splitting the model up between several GPUs would add a third much worse bottleneck – memory bandwidth between the GPUs. No matter how well you connect them, it'll be slower than transfer within a single GPU. Still, the fact that you can fit an 8× larger GPU might be worth it to you. It's a trade-off that's almost universally made while training LLMs (sometimes even with the model split down both its width and length), but is much less attractive for inference.
- amelius 2y ago> Splitting the model up between several GPUs would add a third much worse bottleneck – memory bandwidth between the GPUs. What if you allowed the system to only have a shared memory between every neighboring pair of GPUs? Would that make sense for an LLM?
- imtringued 2y agoAt least for LLMs and transformers this isn't relevant. Having 8x the chips and 8x the memory bandwidth is always better. Interchip communication for matrix multiplication against a constant left matrix with a tiny right matrix isn't bandwidth bound, only latency bound.
- treprinum 2y agoK80 used to be two glued K40 but their interconnect was barely faster than PCIe so it didn't have much benefit as one had to move stuff between two internal GPUs anyway.
- ryao 2y agoInference workloads likely won’t care very much. For llama 3.1 405B with bf16 when you split the workload across GPUs by layer, you need to do a 32KB memory copy before the next GPU can begin processing. That can be done incredibly quickly over PCI-E.
- ryao 2y agoYes: https://news.ycombinator.com/item?id=42313615 https://news.ycombinator.com/item?id=42313615