3 ms·
For training you can often divide the batch size by n (and then only apply the backprop gradient stuff after each n batches for it to be mathematically equivale
by matsemann 4y ago
For training you can often divide the batch size by n (and then only apply the backprop gradient stuff after each n batches for it to be mathematically equivalent). At a cost of speed, though.
- amelius 4y agoDo libraries like torch and tensorflow facilitate this?
- fragmede 4y agoYes, eg https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html https://pytorch.org/docs/stable/generated/torch.nn.parallel....
- amelius 4y agoThank you!
- matsemann 4y agoQuite trivial to implement this yourself if you want to. See gradient accumulation in fastai for instance https://www.kaggle.com/code/jhoward/scaling-up-road-to-the-top-part-3 https://www.kaggle.com/code/jhoward/scaling-up-road-to-the-t...