3 ms·
Quick question: What is a "gradient" in this context? Is it a file? Is it some state that is stored somewhere? My understanding of ML is, I made what is basica
by dinobones 3y ago
Quick question: What is a "gradient" in this context? Is it a file? Is it some state that is stored somewhere?
My understanding of ML is, I made what is basically a "Hot Dog or Not Hot Dog" image classifier and know what a neural net is.
The gradient for that simple neural net, was just found by running Adam an optimizer on the current batch, and then updating the model weights. So by "gradient" do you mean model weights?
- lumost 3y agoThe gradient is the direction and magnitude of change for each model weight. The optimizer determines how to adjust the weight based on the gradient.
- UncleOxidant 3y agoNo, it's an adjustment to model weights that is made during training. Given some input and some expected value there will be some delta and that is used to calculate the gradient - these networks are essentially being trained by gradient descent. As for the size of that data that has to be shared it's going to depend on network size and what kind of representation you're using for the weights - probably bfloat16 these days, but we're certainly seeing a lot of 4 bit representations now.