4 ms·

I am surprised the SGD extensions of https://arxiv.org/abs/1704.04289 https://arxiv.org/abs/1704.04289 have not been picked up yet (via https://www.inference.vc

I am surprised the SGD extensions of https://arxiv.org/abs/1704.04289 https://arxiv.org/abs/1704.04289 have not been picked up yet (via https://www.inference.vc/everything-that-works-works-because-its-bayesian-2/ https://www.inference.vc/everything-that-works-works-because...).

moultano 8y ago
It's interesting work, but I don't think it's relevant to real world models, because real world models violate the conditions they assume at the outset. Namely, real models are not run until anything near convergence on the training set. Everyone uses early stopping to prevent them from overfitting, which they happily do if left to train forever.
- sgt101 8y ago
  Has there been any progress in the art around this issues? The idea of optimise until it stops making sense to optimise is terribly unsatisfying. I think it means "don't optimise this, but move to another state in a way that is approximately optimisation" or "optimise to this special state". Is there art to define "this special state" which is the generalising classifier. There wasn't anything (for sure) in 1997, although we thought that there were two types of risk minimisation - structural risk (size of classifier) and empirical risk (error on datasets) but no one had an idea of how to use these ideas well. I did some work where I pruned optimised ensembles using GA's to get better generalisation vs direct trained, but everyone hated it because it was inefficient. Then people looked at deep networks and the structural risk minimisation stuff went out of the window I guess.
  - beagle3 8y ago
    In simpler worlds, there's a variety of criteria such as Akaike's Information Criterion (AIC) and the Bayesian Information Criterion (BIC). Neither applies directly to NN optimization because they amount to "every additional parameter costs _this_ much". The more philosophical Minimal Description Length (MDL) principles and friends might yield something applicable for NN training, but hasn't yet AFAIK.

moultano 8y ago

It's interesting work, but I don't think it's relevant to real world models, because real world models violate the conditions they assume at the outset. Namely, real models are not run until anything near convergence on the training set. Everyone uses early stopping to prevent them from overfitting, which they happily do if left to train forever.

sgt101 8y ago
Has there been any progress in the art around this issues? The idea of optimise until it stops making sense to optimise is terribly unsatisfying. I think it means "don't optimise this, but move to another state in a way that is approximately optimisation" or "optimise to this special state". Is there art to define "this special state" which is the generalising classifier. There wasn't anything (for sure) in 1997, although we thought that there were two types of risk minimisation - structural risk (size of classifier) and empirical risk (error on datasets) but no one had an idea of how to use these ideas well. I did some work where I pruned optimised ensembles using GA's to get better generalisation vs direct trained, but everyone hated it because it was inefficient. Then people looked at deep networks and the structural risk minimisation stuff went out of the window I guess.
- beagle3 8y ago
  In simpler worlds, there's a variety of criteria such as Akaike's Information Criterion (AIC) and the Bayesian Information Criterion (BIC). Neither applies directly to NN optimization because they amount to "every additional parameter costs _this_ much". The more philosophical Minimal Description Length (MDL) principles and friends might yield something applicable for NN training, but hasn't yet AFAIK.