3 ms·
DistributedDataParallel (potentially) does both model and data parallelism. Data parallelism is also absolutely used when training large models, it has its down
by ImprobableTruth 4y ago
DistributedDataParallel (potentially) does both model and data parallelism. Data parallelism is also absolutely used when training large models, it has its downsides, but I don't think there's any way around it if you're training with a large amount of gpus.
- p1esk 4y agoHow does DDP do model parallel?
- ImprobableTruth 4y agoI phrased that wrongly, DDP itself doesn't of course. I meant that using it in the way GP does is also doing model parallelism.
- p1esk 4y agoI don’t see any mention of model parallel in GP post. How could you possibly use DDP to enable it?
- ImprobableTruth 4y agoGP contrasts DP and DDP by saying that DP is "where you clone your model over each GPU" and DDP is "'proper' multi-GPU training - you can now train big models and put a little bit of data on each GPU". That's simply not what DP or DDP is. What could this possibly mean if it's not misunderstanding DP as data parallelism and DDP as model parallelism? I'm fairly certain that what they're describing is using DDP (which only does data parallelism) in addition to model parallelism.