That's different, NCCL manages traffic between GPUs, but each GPU (with its memory) is still treated as a separate entity by Pytorch or TF. Basically if your mo

That's different, NCCL manages traffic between GPUs, but each GPU (with its memory) is still treated as a separate entity by Pytorch or TF. Basically if your model does not fit in a single GPU memory, Pytorch will not automatically distribute it across two GPUs even if the GPUs are connected with NVLink. You can use model/tensor parallel methods to do this, but it's not going to be automatic (will require writing extra code).