26 ms·
I don’t see any mention of model parallel in GP post. How could you possibly use DDP to enable it?
by p1esk 4y ago
I don’t see any mention of model parallel in GP post. How could you possibly use DDP to enable it?
- ImprobableTruth 4y agoGP contrasts DP and DDP by saying that DP is "where you clone your model over each GPU" and DDP is "'proper' multi-GPU training - you can now train big models and put a little bit of data on each GPU". That's simply not what DP or DDP is. What could this possibly mean if it's not misunderstanding DP as data parallelism and DDP as model parallelism? I'm fairly certain that what they're describing is using DDP (which only does data parallelism) in addition to model parallelism.