4 ms·
This is an exciting work. Gradient noise scale would be interesting to compute individually for each weight/bias. I'm curious how it scales as a function of la
by taliesinb 8y ago
This is an exciting work.
Gradient noise scale would be interesting to compute individually for each weight/bias. I'm curious how it scales as a function of layer depth, for example -- it could well be that different layers require very different treatment in terms of learning rate. Supporting this is that gradient magnitudes are typically very different for different layers, although they evolve similarly in time. Here's something I just whipped up in Mathematica:
https://imgur.com/a/HhCnlDL https://imgur.com/a/HhCnlDL
- samsamoa 8y agoAuthor here— Yes, this is a great question! I think there's a lot more interesting work to be done here. It would be great if we could understand e.g. why layer-wise learning rates help with large-batch training of ImageNet (https://arxiv.org/abs/1708.03888 https://arxiv.org/abs/1708.03888), and maybe the per-component noise scale has something to do with it.