7 ms·
I phrased that wrongly, DDP itself doesn't of course. I meant that using it in the way GP does is also doing model parallelism.
by ImprobableTruth 4y ago
I phrased that wrongly, DDP itself doesn't of course. I meant that using it in the way GP does is also doing model parallelism.
- p1esk 4y agoI don’t see any mention of model parallel in GP post. How could you possibly use DDP to enable it?
- ImprobableTruth 4y agoGP contrasts DP and DDP by saying that DP is "where you clone your model over each GPU" and DDP is "'proper' multi-GPU training - you can now train big models and put a little bit of data on each GPU". That's simply not what DP or DDP is. What could this possibly mean if it's not misunderstanding DP as data parallelism and DDP as model parallelism? I'm fairly certain that what they're describing is using DDP (which only does data parallelism) in addition to model parallelism.