3 ms·

Because augmentation is done on the CPU, and a slower CPU can't keep up if you have 4x 2080ti's in your workstation training in fp16, which is how I prefer to r

by m0zg 6y ago

Because augmentation is done on the CPU, and a slower CPU can't keep up if you have 4x 2080ti's in your workstation training in fp16, which is how I prefer to run things. Moreover, a SATA SSD also can't keep up, so you need NVMe. And for that you need an extension cable, since NVMe stick will overheat in its default location (right under the GPU). Found out the hard way. This is also why AWS sucks so bad for deep learning workloads: on some workloads it's very easy to bottleneck on CPU, and unlike with Google Cloud, you can't just drag the slider and give your VM more cores when needed. Jeff Bezos determined that 4xV100 should get 32 hyperthreads, so you get 32 hyperthreads.

rvz 6y ago
Yes. From the extra context you have given, since you already have multiple high-end GPUs, it makes sense here in order to remove the bottlenecks of a weaker CPU for example. Then I agree on this to some extent. But for those who don't have multiple GPUs and want to do DL training, I'm sure they would start off with one NVIDIA GPU and a more recent CPU that is just good enough for DL training rather than a high-end powerful CPU, which I why I hated to ask this investment-wise.
- m0zg 6y ago
  CPU must not be overlooked, IMO. You don't want to spend thousands of dollars on GPUs and then only be able to keep them busy 50% of the time, if that.
ctchocula 6y ago
Have you looked into using DALI [1] to do augmentation on GPU? They've gotten some nice speedups for computer vision that way. [1] https://devblogs.nvidia.com/fast-ai-data-preprocessing-with-nvidia-dali/ https://devblogs.nvidia.com/fast-ai-data-preprocessing-with-...
- m0zg 6y ago
  Of course, I've looked at everything. Given how expensive modern GPUs are, it is best to use their resources for deep learning rather than augmentation. That way you also get to pipeline augmentation: while GPU is doing the forward/backward pass, the CPU is also cranking out the next batch, so in a way, you're utilizing the resources better. Another issue is, DALI only supports a subset of augmentations that e.g. Albumentations supports, and I'd much rather be working on the "neural" bits than wrestling with augmentation algorithms.

by m0zg 6y ago

rvz 6y ago

Yes. From the extra context you have given, since you already have multiple high-end GPUs, it makes sense here in order to remove the bottlenecks of a weaker CPU for example. Then I agree on this to some extent. But for those who don't have multiple GPUs and want to do DL training, I'm sure they would start off with one NVIDIA GPU and a more recent CPU that is just good enough for DL training rather than a high-end powerful CPU, which I why I hated to ask this investment-wise.

m0zg 6y ago
CPU must not be overlooked, IMO. You don't want to spend thousands of dollars on GPUs and then only be able to keep them busy 50% of the time, if that.

ctchocula 6y ago

Have you looked into using DALI [1] to do augmentation on GPU? They've gotten some nice speedups for computer vision that way. [1] https://devblogs.nvidia.com/fast-ai-data-preprocessing-with-nvidia-dali/ https://devblogs.nvidia.com/fast-ai-data-preprocessing-with-...

m0zg 6y ago
Of course, I've looked at everything. Given how expensive modern GPUs are, it is best to use their resources for deep learning rather than augmentation. That way you also get to pipeline augmentation: while GPU is doing the forward/backward pass, the CPU is also cranking out the next batch, so in a way, you're utilizing the resources better. Another issue is, DALI only supports a subset of augmentations that e.g. Albumentations supports, and I'd much rather be working on the "neural" bits than wrestling with augmentation algorithms.