4 ms·

It's interesting, I once tried injecting perturbations for a short period selectively when the norm of the gradient was near zero, and it gave me consistent imp

It's interesting, I once tried injecting perturbations for a short period selectively when the norm of the gradient was near zero, and it gave me consistent improvement.

ovi256 9y ago
It seems like it would add similar behaviour to simulated annealing. I wonder how this applies to SGD or batch gradient descent, which is commonly used in machine learning applications. Something to try on a toy dataset like MNIST :)
- deepnotderp 9y ago
  How about ImageNet and CIFAR? :)
AndrewOMartin 9y ago
How do you select the average size/standard deviation of your perturbations? Too small and you get no benefit, or very little speed up, too large and you lose the ability to sensibly optimise, and if you attempt to make an adaptive function then you'll find yourself with another postdoc.
- LolWolf 9y ago
  You could probably let the schedule of the perturbations be a sequence ε→0, such that ∑ε = ∞, and such that ∑ηε < ∞ with probability 1, where η is IID, bernoulli on {-1, 1} (e.g. a square-summable sequence will do). I suspect this will be a fairly good start. Note that, then it should in general be independent of the chosen parameter, since the added noise converges in probability to a constant whenever the distribution is close to uniform, but can sum to any real value with consistent bias.
- deepnotderp 9y ago
  It's a scheduled hyperparameter like the learning rate. All my scalar/non-architectural hyperparameters are optimized using DQNs.
cool_username 9y ago
That's generally helpful and usually enough, but it's not universally helpful, depending on the shape of the problem and your step size - see Figure 5 below. (http://graemebell.net/pubs/taros05-bl-embedded-preprint.pdf http://graemebell.net/pubs/taros05-bl-embedded-preprint.pdf) It also depends what the impact of 'perturbations' is in the real world. Random juddering motions can be bad for physical platforms like robots, spacecraft. [I once saw a conference presentation where people were encountering saddle point problems when automatically aligning groups tiny space craft in space] In computer games, random perturbations can make movement look strange, unrealistic or creepy.
- deepnotderp 9y ago
  I should clarify that this is in the context of deep learning optimization (training) only.

It's interesting, I once tried injecting perturbations for a short period selectively when the norm of the gradient was near zero, and it gave me consistent improvement.

ovi256 9y ago

It seems like it would add similar behaviour to simulated annealing. I wonder how this applies to SGD or batch gradient descent, which is commonly used in machine learning applications. Something to try on a toy dataset like MNIST :)

deepnotderp 9y ago
How about ImageNet and CIFAR? :)

AndrewOMartin 9y ago

How do you select the average size/standard deviation of your perturbations? Too small and you get no benefit, or very little speed up, too large and you lose the ability to sensibly optimise, and if you attempt to make an adaptive function then you'll find yourself with another postdoc.

LolWolf 9y ago
You could probably let the schedule of the perturbations be a sequence ε→0, such that ∑ε = ∞, and such that ∑ηε < ∞ with probability 1, where η is IID, bernoulli on {-1, 1} (e.g. a square-summable sequence will do). I suspect this will be a fairly good start. Note that, then it should in general be independent of the chosen parameter, since the added noise converges in probability to a constant whenever the distribution is close to uniform, but can sum to any real value with consistent bias.
deepnotderp 9y ago
It's a scheduled hyperparameter like the learning rate. All my scalar/non-architectural hyperparameters are optimized using DQNs.

cool_username 9y ago

That's generally helpful and usually enough, but it's not universally helpful, depending on the shape of the problem and your step size - see Figure 5 below. (http://graemebell.net/pubs/taros05-bl-embedded-preprint.pdf http://graemebell.net/pubs/taros05-bl-embedded-preprint.pdf) It also depends what the impact of 'perturbations' is in the real world. Random juddering motions can be bad for physical platforms like robots, spacecraft. [I once saw a conference presentation where people were encountering saddle point problems when automatically aligning groups tiny space craft in space] In computer games, random perturbations can make movement look strange, unrealistic or creepy.

deepnotderp 9y ago
I should clarify that this is in the context of deep learning optimization (training) only.