4 ms·

$4800 is what the previous RTX A6000 costs in the US after taxes. That is a BIG cost for anyone outside of a professional setting. You could buy 2 x 3090's wit

$4800 is what the previous RTX A6000 costs in the US after taxes. That is a BIG cost for anyone outside of a professional setting. You could buy 2 x 3090's with NVLINK for half that. I recently bought a datacenter M40 with 24GB of RAM for $150 on Ebay then added water cooling. The Dell Precision 5810 I bought second hand can now run some really interesting stuff on those 24GB of RAM albeit slower than a 3090. For less than $500 I can have two watercooled M40's that will run almost anything I can throw at them.

bitL 4y ago
2x3090 with NVLink isn't appearing as a single GPU even on Linux.
- monkmartinez 4y ago
  My understanding may be flawed, but I am under the impression that 2 x GPUs with NVLink will pool memory. It does not change the underlying 2 x GPU architecture in a way that would represent 2x as a single when looking at the system config. Furthermore, I believe the application code needs to have NVLink enabled to take advantage of the pooled memory. Pytorch and Tensorflow have knobs to turn in order to take advantage of NVLink... where programs like DaVinci Resolve may need patching, etc.
  - p1esk 4y ago
    2 x GPUs with NVLink will pool memory None of major ML frameworks such as Pytorch of TF support that. I’m not sure why.
    - lostmsu 4y ago
      I think they both support data exchange over the NVLink via NCCL.
      - p1esk 4y ago
        That's different, NCCL manages traffic between GPUs, but each GPU (with its memory) is still treated as a separate entity by Pytorch or TF. Basically if your model does not fit in a single GPU memory, Pytorch will not automatically distribute it across two GPUs even if the GPUs are connected with NVLink. You can use model/tensor parallel methods to do this, but it's not going to be automatic (will require writing extra code).
maldev 4y ago
I've been thinking of doing the same with the M40. I'm more interested in learning and time isn't really an issue, mainly just running big models. Is water cooling that important? Is it great for a prosumer looking to run models and research the field with?

bitL 4y ago

2x3090 with NVLink isn't appearing as a single GPU even on Linux.

monkmartinez 4y ago
My understanding may be flawed, but I am under the impression that 2 x GPUs with NVLink will pool memory. It does not change the underlying 2 x GPU architecture in a way that would represent 2x as a single when looking at the system config. Furthermore, I believe the application code needs to have NVLink enabled to take advantage of the pooled memory. Pytorch and Tensorflow have knobs to turn in order to take advantage of NVLink... where programs like DaVinci Resolve may need patching, etc.
- p1esk 4y ago
  2 x GPUs with NVLink will pool memory None of major ML frameworks such as Pytorch of TF support that. I’m not sure why.
  - lostmsu 4y ago
    I think they both support data exchange over the NVLink via NCCL.
    - p1esk 4y ago
      That's different, NCCL manages traffic between GPUs, but each GPU (with its memory) is still treated as a separate entity by Pytorch or TF. Basically if your model does not fit in a single GPU memory, Pytorch will not automatically distribute it across two GPUs even if the GPUs are connected with NVLink. You can use model/tensor parallel methods to do this, but it's not going to be automatic (will require writing extra code).

maldev 4y ago

I've been thinking of doing the same with the M40. I'm more interested in learning and time isn't really an issue, mainly just running big models. Is water cooling that important? Is it great for a prosumer looking to run models and research the field with?