3 ms·
2 x GPUs with NVLink will pool memory None of major ML frameworks such as Pytorch of TF support that. I’m not sure why.
by p1esk 4y ago
2 x GPUs with NVLink will pool memory
None of major ML frameworks such as Pytorch of TF support that. I’m not sure why.
- lostmsu 4y agoI think they both support data exchange over the NVLink via NCCL.
- p1esk 4y agoThat's different, NCCL manages traffic between GPUs, but each GPU (with its memory) is still treated as a separate entity by Pytorch or TF. Basically if your model does not fit in a single GPU memory, Pytorch will not automatically distribute it across two GPUs even if the GPUs are connected with NVLink. You can use model/tensor parallel methods to do this, but it's not going to be automatic (will require writing extra code).