3 ms·
DGX Sparks have 200Gbps interlinks for clustering. Apparently switches are the only practical way to scale such clusters beyond 2 nodes even though each Spark h
by hgoel 1mo ago
DGX Sparks have 200Gbps interlinks for clustering. Apparently switches are the only practical way to scale such clusters beyond 2 nodes even though each Spark has 2 such ports.
- FireBeyond 1mo agoWell, the problem is that you want each node to be able to directly connect to each other, so you run into a mesh situation. https://www.youtube.com/watch?v=QJqKqxQR36Y https://www.youtube.com/watch?v=QJqKqxQR36Y shows 8 running at 100gbps and clustered.
- hgoel 1mo agoI understand that part, what I don't quite get is why we can't divide the layers such that each node only needs to talk to at most 2 nodes. I assume I'm misunderstanding how work is distributed between them, but haven't looked into the details yet.
- wtallis 1mo agoI think this is the distinction between pipeline parallelism (first N layers on machine 1, next N layers on machine 2, etc.) vs tensor parallelism where all machines contribute to the same matrix multiplication operation within the same layer. Pipeline parallelism doesn't need exotic high speed networking, but also doesn't deliver the kind of speed increase that tensor parallelism provides.