3 ms·

Technically, it is no where near that. I'm not even sure what a "full set of gradients" for an image would even mean - it's not a term. Weights are usually ran

Technically, it is no where near that. I'm not even sure what a "full set of gradients" for an image would even mean - it's not a term. Weights are usually randomly initialized, defining a function. Many images (which are randomly batched on each epoch) are passed through, and a loss function tallied. Backprop gets a gradient, and a training schedule tweaks the weight a little bit. No image has its "gradients" somehow added. The overall function has gradients, but they are not smooth, due to many things such as ReLU being non-smooth. These are not invertible in any sense. Th networks have many, many irreversible steps in them that truncate, that max pool, that perform dropout during steps, many of these choices done randomly. There are gradient normalizations to clip or stretch to deal with vanishing or exploding gradients, and SOOO many lossy things done that are completely irreversible. So none of this can possibly work. A network is not simply a sum of images in some format.

visarga 4y ago
For each training example gradients are computed, then averaged per batch, then subtracted from model weights in proportion to the learning rate. So each example generates a "full set of gradients" (one incremental change) for all the weights of the model. The model doesn't save the image itself, but eats the gradients. The question is how we see this process, does it entail a copyright penalty?
- ChrisLomont 4y ago
  >For each training example gradients are computed, then averaged per batch No, that would be incredibly slow. The loss is summed over items in a batch, then the gradient is computed once based on the loss, then backpropagation is done. It's done this way by default both in TensorFlow and in Pytorch. It is possible to sum per entry (see here [1] in pytorch docs) but is is extremely rare. The majority of training loops (this one included) is the standard "do a batch" then call "loss.backward()" which computes only once per batch. Look at the "backward()" call in pytorch docs [2]: This "Computes the gradient of current tensor w.r.t. graph leaves." If this were done per item and being summed, there would be no need for this in the training loop, and nearly every training loop I've seen has this. Here's pytorch explaining all this in detail [3]. And in any case, since the functions in the pipeline contain many non-smooth terms, the information required to restore an image, even if only one were done, is not in the gradient, any more than the slope of a line tells you the x-intercept. Another way to think of it: even for perfectly smooth functions, derivatives lose information that cannot be recovered. [1] https://pytorch.org/functorch/stable/notebooks/per_sample_grads.html https://pytorch.org/functorch/stable/notebooks/per_sample_gr... [2] https://pytorch.org/docs/stable/generated/torch.Tensor.backward.html https://pytorch.org/docs/stable/generated/torch.Tensor.backw... [3] https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html https://pytorch.org/tutorials/beginner/blitz/autograd_tutori...
  - visarga 4y ago
    Back-propagating a batch you go from one loss to the sum of individual losses and then down to each example separately. Each example has different activations that multiply with the upstream gradients, so first we need to compute per example weight gradients. The computation is a perfect mirror of the forward pass.

visarga 4y ago

For each training example gradients are computed, then averaged per batch, then subtracted from model weights in proportion to the learning rate. So each example generates a "full set of gradients" (one incremental change) for all the weights of the model. The model doesn't save the image itself, but eats the gradients. The question is how we see this process, does it entail a copyright penalty?

ChrisLomont 4y ago
>For each training example gradients are computed, then averaged per batch No, that would be incredibly slow. The loss is summed over items in a batch, then the gradient is computed once based on the loss, then backpropagation is done. It's done this way by default both in TensorFlow and in Pytorch. It is possible to sum per entry (see here [1] in pytorch docs) but is is extremely rare. The majority of training loops (this one included) is the standard "do a batch" then call "loss.backward()" which computes only once per batch. Look at the "backward()" call in pytorch docs [2]: This "Computes the gradient of current tensor w.r.t. graph leaves." If this were done per item and being summed, there would be no need for this in the training loop, and nearly every training loop I've seen has this. Here's pytorch explaining all this in detail [3]. And in any case, since the functions in the pipeline contain many non-smooth terms, the information required to restore an image, even if only one were done, is not in the gradient, any more than the slope of a line tells you the x-intercept. Another way to think of it: even for perfectly smooth functions, derivatives lose information that cannot be recovered. [1] https://pytorch.org/functorch/stable/notebooks/per_sample_grads.html https://pytorch.org/functorch/stable/notebooks/per_sample_gr... [2] https://pytorch.org/docs/stable/generated/torch.Tensor.backward.html https://pytorch.org/docs/stable/generated/torch.Tensor.backw... [3] https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html https://pytorch.org/tutorials/beginner/blitz/autograd_tutori...
- visarga 4y ago
  Back-propagating a batch you go from one loss to the sum of individual losses and then down to each example separately. Each example has different activations that multiply with the upstream gradients, so first we need to compute per example weight gradients. The computation is a perfect mirror of the forward pass.