11 ms·
> will this algorithm uncover a new design space for NNs? No. Hinton "discovered" stacking ensembles and gave it a new name, fancy analogies to biological bra
by PartiallyTyped 4y ago
> will this algorithm uncover a new design space for NNs?
No.
Hinton "discovered" stacking ensembles and gave it a new name, fancy analogies to biological brains and then made it worse.
The gist of this is that you can select a computational unit, be it a linear layer, or a collection of layers, compute the derivative of the output with respect to the parameters, and update them.
Each computational unit is independent, meaning that you don't calculate gradients going outside of it.
This is the same as training a bunch of networks, computing predictions, and then using another layer to combine the predictions. This is called stacking, and the networks are called an "ensemble". You can do this multiple times and have N levels of meta estimators.
Instead of fitting the ensemble and then the meta estimator, Hinton proposes training both simultaneously but without allowing gradients to flow through.
That is stupid because if you don't allow gradients to flow through, you will see a context drift as the data distribution changes. Hinton observed this context drift, to deal with that, he proposed normalizing the data.
On one extreme, you can use individual linear units as the models, and on the other extreme, you can combine all units into a single neural network and treat that as a module.
So no, this does not open any new design, it's an old idea, worsened, and wrapped in fancy words and post-facto reasoning.
If you are curious how a linear layer is an ensemble, observe that each vector is its own linear estimator, making the linear mapping an ensemble of estimators.
- sdwr 4y agoThats a lot less cheaty, biologically speaking, than full backprop. This Hinton guy sounds like he knows what he's talking about. "Context drift as data distribution changes" sounds a hell of a lot like real life to me. Normalized = hedonic treadmill on long view At the micro scale, data that overflows the normalization is stored in emotional state, creating an orthogonal source of truth that makes up for the lack of full connected learning.
- chamwislothe2nd 4y agoYou're accusing one of the foundations of modern AI with either being a fraud, or incompetent. At best that seems short sighted, no?
- nayroclade 4y agoYou're obviously new to Hacker News :-D
- chamwislothe2nd 4y agoUnfortunately, I've been here for a decade in one form or another. Every now and then someone writes something so pompous that I just can't help myself but post. Back to lurking now. Cheers!
- PartiallyTyped 4y agoIf we can't publicly scrutinize people who have great sway in the industry, what does that say about us as a research community? The fact that I argued why I found it bogus based on well established principles, and I get shitted on by people who by all means have provided nothing to this conversation and except suppressing criticism or throwing ad-hominems should tell all about the quality of discourse. Dismissing criticism, not by arguments, but by the mere name of the person does a disservice to everyone. If the research can't stand on its own, independent of the author, then it is not good research.
- chamwislothe2nd 4y agoWe are all looking forward to your research paper that disproves his claims. Or you know, any proof.
- PartiallyTyped 4y agoI argued for it and all I got was downvoted without criticism of the substance of my arguments, only ad-hominems and fallacies. If you can point to _fundamental_ criticism of my arguments, and not fallacies or attacks, I'd be more than happy to discuss them.
- chamwislothe2nd 4y ago
- hgsgm 4y ago[flagged]
- PartiallyTyped 4y agoAd-hominems are not a particularly nice way to argue about correctness of a claim.
- NiloCK 4y agoMaybe you know something but I don't, but I believe the comment you are replying to was a compliment and not a sarcastic dig.
- PartiallyTyped 4y agoThis person has made a similar remark and purposefully looked into my post history - which doesn't make any claims about my knowledge or skills - lied about it, and made a sarcastic remark, see "everything".
- randomdata 4y agoA lot can change in 60 days.
- ekleraki 4y agoThey do have a master's degree according to the post. The claims made are not that deep for researchers.
- belter 4y agoThe use of the word masters is now considered not cool according to Stanford... :-)
- PartiallyTyped 4y agoIt's not like I have been doing many things outside reading papers and books over the past few years... The post that person used as an ad-hominem even says so.
- dimatura 4y agoYeah, no. Reading the paper I don't really see anything but a superficial resemblance to stacking. Hinton was active back when Wolpert introduced stacking and I'm fairly sure he is aware of it. If anything it much more closely resembles his own prior work in Boltzmann machines, unsurprisingly (and which he cites), or even his prior work on capsules. I don't know if this will really pan out into anything that different or useful for the field, but it's unfair and inaccurate to dismiss it as derivative of stacking.
- ekleraki 4y agoA single linear layer is for all intents and purposes equivalent to running an ensemble of linear estimators. By disallowing gradients to flow between two layers A, B, computing (B . f . A)(x) with f being a non linearity, the second layer is an ensemble of linear estimators of the outputs of the first, and for all intents and purposes, the output of (f.A)(x) is just preprocessing for B. Since gradients don't flow from B to A in (B.f.A)(x), A is trained independently of B, meaning that the training distribution of B changes without B influencing it, i.e. context drift. B doesn't know the difference, and B doesn't influence it. For all intents and purposes, you can compute all the outputs as training of A happens, meaning training A to completion, and then feed them into B and B will still compute the same outputs and derivatives as it did before. To deal with context drift, Hinton proposes normalizing the data, so the distribution does not change significantly. Whatever he proposed is not "backprop-free" either. It still involves backprop, but the number of layers gradients flow through is 1, the layer itself. The argument that you can still train through non-differentiable operations is not particularly convincing either; the reparameterization trick shows that is trivial to pass gradients through non differentiable operations if we are smart about it. Given non differentiable operator Z: R^N -> R^N; let A, B, C be R^N -> R^N linear layer, B(Z(C(x)) * A(C(x))) allows gradients to flow through B and A all the way to C. The output of Z is for all intents and purposes a Hadamard product with (A . C)(x) that is runtime constructed and might as well be part of the input. You can even run Z(C(x)) through a neural network and learn how to transform that and still provide useful and informative gradients back to C(x) via (A . C)
- dimatura 4y ago