4 ms·
It seems that the point is that the objective function is applied layerwise, still computes gradient to get the update direction, it's just that gradients don't
by PartiallyTyped 4y ago
It seems that the point is that the objective function is applied layerwise, still computes gradient to get the update direction, it's just that gradients don't propagate to previous layers (detatched tensor).
As far as I can tell, this is almost the same as stacking multiple layers of ensembles, except worse as each ensemble is trained while previous ensembles are learning. This is causing context drift.
To deal with the context drift, Hinton proposes to normalise the output.
This isn't anything new or novel. Expressing "ThIs LoOkS sImIlAr To HoW cOgNiTiOn WoRkS" to make it sound impressive doesn't make it impressive or good by any stretch of the imagination.
Hinton just took something that existed for a long time, made it worse, gave it a different name and wrapped it in a paper under his name.
With every paper I am more convinced that the Laureates don't deserve the award.
Sorry, this "paper" smells from a mile away, and the fact that it is upvoted as much shows that people will upvote anything if they see a pretty name attached.
Edit:
Due to the apparent controversy of my criticism, I can't respond with a reply, so here is my response to the comment below asking what exactly makes this worse.
> As far as I can tell, this is almost the same as stacking multiple layers of ensembles
It isn't new. Ensembling is used and has been used for a long time. All kaggle competitions are won through ensembles and even ensembles of ensembles. It is a well studied field.
> except worse as each ensemble is trained while previous ensembles are learning.
Ensembles exhibit certain properties, but only iff they are trained independently from each other. This is well studied, you can read more about it in Bishop's Pattern recognition book.
> This is causing context drift.
Context drift occurs when a distribution changes over time. This changes the loss landscape which means the global minima change / move.
> To deal with the context drift, Hinton proposes to normalise the output.
So not only is what Hinton built a variation of something that existed already, made it worse by training the models simultaneously, and to handle the fact that it is worse, he adds additional computations to deal with said issue.
- hgsgm 4y agoThis command is a lot of words to say "I don't like it" without giving any reason to believe you. If it's not new or novel, why aren't people using it? If it's bad, what's wrong with it?
- LudwigNagasena 4y agoIt performs worse than baсkprop.
- qorrect 4y agoIs it slower and less accurate ? Or just slower.
- LudwigNagasena 4y agoIt is far less accurate compared to SOTA models. The paper says it should train faster, but it doesn't provide any metrics; so it's hard to make any "pound for pound" comparisons.
- UncleEntity 4y agoSo the initial “hey, this might be a good idea” implementation performs slightly worse than something that has had literally billions of dollars thrown at it?
- ekleraki 4y agoIt is slower to the same backprop that we have used for decades now. No comparisons to AdamW were made. In fact, this algorithm uses backprop at its core, but propagating through 0 layers.
- LudwigNagasena 4y agoThe question was "If it's not new or novel, why aren't people using it?". For example, take a look at this paper: https://arxiv.org/abs/1905.11786 https://arxiv.org/abs/1905.11786. It was published 3 years ago, it also does parallel layer-wise optimization, it even talks about being inspired by biology, though the objective function is different from Hinton's. Why aren't people using it? Because it performs worse. It is that simple. Is it an interesting area to explore? Probably. There are millions of interesting areas to explore. It doesn't mean it is worth using, at least yet.
- 4y ago
- goethes_kind 4y agoSince you seem to understand what he is saying, can you explain to me how the per layer objective function looks like? I don't get what he means by inserting the label into the input and what labels he is using per layer.
- LudwigNagasena 4y agoA model is a function F that minimizes error in y_i = F(X_i) + error. Inserting a label simply means a function F(X_i,y_j). Then you optimize it in some way to separate true labels from false label, e.g. F(X_i, y_j) = (y_i==y_j)-(y_i!=y_j) + error.
- bjourne 4y agoYou train the network to detect correlations between the values of the ten first pixels and the rest of the image. Imagine you have a bunch of images of digits. For images with digit three you set the third pixel to white, for images of the digit four, you set the fourth pixel to white, and so on (actually, zero-indexing so fourth and fifth pixel for digit three and four but whatever). The other nine pixels among the first ten you set to black. These are positive samples and training the network with them will make it output a big number when it encounters them. Then you swap the pixels so that the images with the digit three has the fourth pixel set to white and the images with the digit four has the third pixel set to white. These are negative samples and they cause the network to output a small number. Thus, the only difference between positive and negative samples is the location of the white pixel. So for an image you want to classify you run it through the network ten times and each time shifting the location of the white pixel. The location for which the network outputs the biggest number is the predicted class. Obviously, this method is problematic if you have thousands of labels or if your network is not a classifier.
- LudwigNagasena 4y agoThat's the same impression I had. I was afraid I am not getting something or missing a bigger picture. I am glad I am not the only one who feels that way.
- deleted 4y ago[deleted]