>For each training example gradients are computed, then averaged per batch No, that would be incredibly slow. The loss is summed over items in a batch, then th

>For each training example gradients are computed, then averaged per batch No, that would be incredibly slow. The loss is summed over items in a batch, then the gradient is computed once based on the loss, then backpropagation is done. It's done this way by default both in TensorFlow and in Pytorch. It is possible to sum per entry (see here [1] in pytorch docs) but is is extremely rare. The majority of training loops (this one included) is the standard "do a batch" then call "loss.backward()" which computes only once per batch. Look at the "backward()" call in pytorch docs [2]: This "Computes the gradient of current tensor w.r.t. graph leaves." If this were done per item and being summed, there would be no need for this in the training loop, and nearly every training loop I've seen has this. Here's pytorch explaining all this in detail [3]. And in any case, since the functions in the pipeline contain many non-smooth terms, the information required to restore an image, even if only one were done, is not in the gradient, any more than the slope of a line tells you the x-intercept. Another way to think of it: even for perfectly smooth functions, derivatives lose information that cannot be recovered. [1] https://pytorch.org/functorch/stable/notebooks/per_sample_grads.html https://pytorch.org/functorch/stable/notebooks/per_sample_gr... [2] https://pytorch.org/docs/stable/generated/torch.Tensor.backward.html https://pytorch.org/docs/stable/generated/torch.Tensor.backw... [3] https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html https://pytorch.org/tutorials/beginner/blitz/autograd_tutori...