It's a fair question. Nvidia's NCCL and AMD's RCCL provide parallelism constructs that really are hidden at the framework level (such as PyT). However, I don'

It's a fair question. Nvidia's NCCL and AMD's RCCL provide parallelism constructs that really are hidden at the framework level (such as PyT). However, I don't think that you would want to hide model, data, or tensor parallelism. It's too important a consideration for performance and training convergence impact. At least in scientific computing, I've never observed effective means of automatic parallelism expressed across many nodes despite decades of research. I'm not optimistic this will be effective anytime soon.