4 ms·

Good article! But the comment on NVLink / PCIe4 (and PCIe3 x4 is good enough) doesn’t fit my experience. PCIe3 x4 can really impact your all-reduce performance

by liuliu 6y ago

Good article! But the comment on NVLink / PCIe4 (and PCIe3 x4 is good enough) doesn’t fit my experience. PCIe3 x4 can really impact your all-reduce performance for models such as Transformers. For ResNet, it can impact a bit too, but to the 5% to 10% range like the author mentioned. I am also interested in whether PCIe4 can help unified memory for larger models. Guess have to wait for RTX 3090 actual release.

p1esk 6y ago
unified memory for larger models What are you talking about? Are you writing custom CUDA code to run those large models? Because there's zero support for unified memory in any of the existing DL frameworks.
- liuliu 6y ago
  Yes. I have an alternative DL framework to play whatever I want with :)
  - p1esk 6y ago
    Link?
    - lunixbochs 6y ago
      Probably this: https://libnnc.org https://libnnc.org https://github.com/liuliu/ccv/tree/unstable/lib/nnc https://github.com/liuliu/ccv/tree/unstable/lib/nnc
lunixbochs 6y ago
I agree. When training production speech recognition models (wav2letter) on huge datasets, I hit PCIe, NVLink, network, and probably storage bottlenecks when scaling. (This isn’t comparable to training on 40GB of OpenWebText, it’s terabytes of compressed audio) When I tested the DGX2, 16-card bandwidth was finally not an issue (I was extremely happy with the NVSwitch)... but I hit a CPU bottleneck going from [64vcpu + 8 V100] to [96vcpu + 16 V100] (even with tricks to reduce CUDA CPU load). I’m excited to sometime try my workload on 2x NVLinked 3090s with a 64-core AMD CPU, optimized PCIe v4 NVMe data path, additional caching, and some hand tuning of the training code. It’s possible this will be competitive with an 8x V100 google cloud instance based on some of my scaling pain graphs. I’d consider 4x 3090s- but between only having a single NVLink port, and having to figure out how to even fit four 3x cards in a case without water cooling, it seems prudent to start with two.

by liuliu 6y ago

p1esk 6y ago

unified memory for larger models What are you talking about? Are you writing custom CUDA code to run those large models? Because there's zero support for unified memory in any of the existing DL frameworks.

liuliu 6y ago
Yes. I have an alternative DL framework to play whatever I want with :)
- p1esk 6y ago
  Link?
  - lunixbochs 6y ago
    Probably this: https://libnnc.org https://libnnc.org https://github.com/liuliu/ccv/tree/unstable/lib/nnc https://github.com/liuliu/ccv/tree/unstable/lib/nnc

lunixbochs 6y ago

I agree. When training production speech recognition models (wav2letter) on huge datasets, I hit PCIe, NVLink, network, and probably storage bottlenecks when scaling. (This isn’t comparable to training on 40GB of OpenWebText, it’s terabytes of compressed audio) When I tested the DGX2, 16-card bandwidth was finally not an issue (I was extremely happy with the NVSwitch)... but I hit a CPU bottleneck going from [64vcpu + 8 V100] to [96vcpu + 16 V100] (even with tricks to reduce CUDA CPU load). I’m excited to sometime try my workload on 2x NVLinked 3090s with a 64-core AMD CPU, optimized PCIe v4 NVMe data path, additional caching, and some hand tuning of the training code. It’s possible this will be competitive with an 8x V100 google cloud instance based on some of my scaling pain graphs. I’d consider 4x 3090s- but between only having a single NVLink port, and having to figure out how to even fit four 3x cards in a case without water cooling, it seems prudent to start with two.