4 ms·
Do libraries like torch and tensorflow facilitate this?
by amelius 4y ago
Do libraries like torch and tensorflow facilitate this?
- fragmede 4y agoYes, eg https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html https://pytorch.org/docs/stable/generated/torch.nn.parallel....
- amelius 4y agoThank you!
- matsemann 4y agoQuite trivial to implement this yourself if you want to. See gradient accumulation in fastai for instance https://www.kaggle.com/code/jhoward/scaling-up-road-to-the-top-part-3 https://www.kaggle.com/code/jhoward/scaling-up-road-to-the-t...