3 ms·
Anyone with experience running 2 linked consumer GPU's want to chime in how good this works in practice?
by PeterStuer 3y ago
Anyone with
experience running 2 linked consumer GPU's want to chime in how good this works in practice?
- Filligree 3y agoYou get a fast link between the GPUs, which should help when you’ve got a model split between them. However, that split isn’t automatic. You can’t expect to run a 40GB model on that, unless perhaps if it’s been designed for that—the way llama.cpp can split a model between the GPU and CPU, for instance. What you can do without trouble is keep more models loaded, do more things at the same time, and occasionally run the same model at double speed if it batches well.
- deaddodo 3y agoCUDA multi-GPU with NVLink is pretty well tested with shared memory space. You still want to use NCCL to optimize the allocation, but many CUDA-aware libraries (and their subsequent ML tools) are capable.
- pseg134 3y agoThis is incorrect if you are talking about 3090 or 3090ti using nvlink.
- PeterStuer 3y agoYou mean those would work like a virtual single GPU with 48GB vram?
- Tepix 3y agoNo. But pytorch will automatically make use of both GPUs and a NVlink bridge if you use its model parallel and distributed data parallel approaches.
- marcyb5st 3y agoI think you need enterprise grade cards for it to make it work. If I remember correctly consumer cards with nvlink can't share resources to host a 40GB model in vram.