3 ms·
I doubt the RAM is added up. I think that’s only a feature reserved for their NVLinked HPC series cards. In fact, without nvlink, I don’t see how you’d connect
by Bancakes 2y ago
I doubt the RAM is added up. I think that’s only a feature reserved for their NVLinked HPC series cards. In fact, without nvlink, I don’t see how you’d connect them together to compute a single task in a performant and efficient way.
- whimsicalism 2y agoyeah essentially this
- minimaxir 2y agoIt depends on how the parallelism is implemented, e.g. distributed data parallel (DDP) to synchronize gradients: https://pytorch.org/tutorials/intermediate/ddp_tutorial.html https://pytorch.org/tutorials/intermediate/ddp_tutorial.html It's a rabbit hole I stay away from for pragmatic reasons.