24 ms·
Geoffrey Hinton publishes new deep learning algorithm
- mdp2021 4y agoDirect link to an implementation on GitHub: https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/forward_forward https://github.com/nebuly-ai/nebullvm/tree/main/apps/acceler... -- The divulgational title is almost an understatement: the Forward-Forward algorithm is an alternative to backpropagation. Edit: sorry, the previous formulation of the above in this post, relative to the advantages, was due to a misreading. Hinton writes: > The Forward-Forward algorithm (FF) is comparable in speed to backpropagation but has the advantage that it can be used when the precise details of the forward computation are unknown. It also has the advantage that it can learn while pipelining sequential data through a neural network without ever storing the neural activities or stopping to propagate error derivatives....The two areas in which the forward-forward algorithm may be superior to backpropagation are as a model of learning in cortex and as a way of making use of very low-power analog hardware without resorting to reinforcement learning
- deleted 4y ago[deleted]
- kumarvvr 4y agoThis is an interesting approach and I have read that this is more closer to how our brains works. We extract learning, while we are imbibing the data and there seems to be no mechanism in the brain that favors backprop like learning process.
- bayesian_horse 4y agoFact: Geoffrey Hinton has discovered how the brain works. Every few years actually.
- rytill 4y agoPaper: https://www.cs.toronto.edu/~hinton/FFA13.pdf https://www.cs.toronto.edu/~hinton/FFA13.pdf
- BenoitP 4y agoNot a deep learning expert, but: it seems that without backpropagation for model updates, the communication costs should be lower. And that will enable models that are easier to parallelize? Nvidia isn't creating new versions of its NVLink/NVSwitch products just for the sake of it, better communication must be a key enabler. Can someone with deeper knowledge can comment on this? Is communication a bottleneck, and will this algorithm uncover a new design space for NNs?
- jerpint 4y agoIt’s more that without backpropagation, you no longer need to store your forward activations across many layers to compute the backwards pass which usually is dependent on a forward pass. When a network is hundreds of layers, and batches are very large, the forwards and backwards accumulations add up in terms of memory required. Communication across GPUs doesn’t solve this but instead allows to have either many models running in parallel on different GPUs to increase Barch size or to share many layers across GPUs to increase model size. Quick communication is critical to maintain training speeds that aren’t astronomical
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- PartiallyTyped 4y ago> will this algorithm uncover a new design space for NNs? No. Hinton "discovered" stacking ensembles and gave it a new name, fancy analogies to biological brains and then made it worse. The gist of this is that you can select a computational unit, be it a linear layer, or a collection of layers, compute the derivative of the output with respect to the parameters, and update them. Each computational unit is independent, meaning that you don't calculate gradients going outside of it. This is the same as training a bunch of networks, computing predictions, and then using another layer to combine the predictions. This is called stacking, and the networks are called an "ensemble". You can do this multiple times and have N levels of meta estimators. Instead of fitting the ensemble and then the meta estimator, Hinton proposes training both simultaneously but without allowing gradients to flow through. That is stupid because if you don't allow gradients to flow through, you will see a context drift as the data distribution changes. Hinton observed this context drift, to deal with that, he proposed normalizing the data. On one extreme, you can use individual linear units as the models, and on the other extreme, you can combine all units into a single neural network and treat that as a module. So no, this does not open any new design, it's an old idea, worsened, and wrapped in fancy words and post-facto reasoning. If you are curious how a linear layer is an ensemble, observe that each vector is its own linear estimator, making the linear mapping an ensemble of estimators.
- maurits 4y agoDeep dive tutorial for learning in a forward pass [1] [1] https://amassivek.github.io/sigprop https://amassivek.github.io/sigprop
- cochne 4y ago> There are many choices for a loss L (e.g. gradient, Hebbian) and optimizer (e.g. SGD, Momentum, ADAM). The output(), y, is detailed in step 4 below. I don't get it, don't all of those optimizers work via backprop?
- mkaic 4y agoThe optimizers take parameters and their gradients as inputs and apply update rules to them, but the gradients you supply can come from anywhere. Backdrop is the most common way to assign gradients to parameters, but other methods can work too—as long as the optimizer is getting both parameters and gradients, it doesn't care where they come from.
- PartiallyTyped 4y agoIt seems that the point is that the objective function is applied layerwise, still computes gradient to get the update direction, it's just that gradients don't propagate to previous layers (detatched tensor). As far as I can tell, this is almost the same as stacking multiple layers of ensembles, except worse as each ensemble is trained while previous ensembles are learning. This is causing context drift. To deal with the context drift, Hinton proposes to normalise the output. This isn't anything new or novel. Expressing "ThIs LoOkS sImIlAr To HoW cOgNiTiOn WoRkS" to make it sound impressive doesn't make it impressive or good by any stretch of the imagination. Hinton just took something that existed for a long time, made it worse, gave it a different name and wrapped it in a paper under his name. With every paper I am more convinced that the Laureates don't deserve the award. Sorry, this "paper" smells from a mile away, and the fact that it is upvoted as much shows that people will upvote anything if they see a pretty name attached. Edit: Due to the apparent controversy of my criticism, I can't respond with a reply, so here is my response to the comment below asking what exactly makes this worse. > As far as I can tell, this is almost the same as stacking multiple layers of ensembles It isn't new. Ensembling is used and has been used for a long time. All kaggle competitions are won through ensembles and even ensembles of ensembles. It is a well studied field. > except worse as each ensemble is trained while previous ensembles are learning. Ensembles exhibit certain properties, but only iff they are trained independently from each other. This is well studied, you can read more about it in Bishop's Pattern recognition book. > This is causing context drift. Context drift occurs when a distribution changes over time. This changes the loss landscape which means the global minima change / move. > To deal with the context drift, Hinton proposes to normalise the output. So not only is what Hinton built a variation of something that existed already, made it worse by training the models simultaneously, and to handle the fact that it is worse, he adds additional computations to deal with said issue.
- hgsgm 4y agoThis command is a lot of words to say "I don't like it" without giving any reason to believe you. If it's not new or novel, why aren't people using it? If it's bad, what's wrong with it?
- 4y ago
- sheerun 4y agoIsn't this similar to how GAN networks learn? Edit: Yes, there is small chapter in paper comparing it to GAN
- tjungblut 4y agoInteresting, my first take was that it's like contrastive divergence in Restricted Boltzmann Machines (RBMs). There's also a chapter for that.
- lvl102 4y agoQuantum would absolutely change everything in DL/ML space.
- Kalanos 4y agoMost problems are less computationally intensive than one would think
- sillysaurusx 4y agoIt turns out there’s a whole subfield for quantum ML. I don’t know much about it, but it’s neat that there’s any applicability. It’s not obvious that there was any connection.
- bayesian_horse 4y agoI remember a long time ago there was a collection of Geoffrey Hinton facts. Like "Geoffrey Hinton once wrote a neural network that beat Chuck Norris."
- mousetree 4y agoI saw the same for Jeff Dean
- goethes_kind 4y agoI skimmed through the paper and am a bit confused. There's only one equation and I feel like he rushed to publish a shower thought without even bothering to flesh it out mathematically. So how do you optimize a layer? Do you still use gradient descent? So you are have a per layer loss with a positive and negative component and then do gradient descent? So then what is the label for each layer? Do you use the same label for each layer? And what does he mean by the forward pass not being fully known? I don't get this application of the blackbox between layers. Why would you want to do that?
- harveywi 4y agoThose details have to be omitted from manuscripts in order to avoid having to cite the works of Jürgen Schmidhuber.
- bitL 4y agoJürgen did it all before in the 80s, however it was never translated to English so Geoffrey could happily reinvent it.
- godelski 4y agoJürgen invented AGI in the early 90s but someone pressed the red button on his website and it committed suicide.
- manytree5 4y agoJust curious, why would one want to avoid citing Schmidhuber's work?
- sva_ 4y agoIt is a bit of a meme in AI research, as Schmidhuber often claims that he hasn't received the citations that he thinks he deserves. https://www.urbandictionary.com/define.php?term=schmidhuber https://www.urbandictionary.com/define.php?term=schmidhuber
- rsfern 4y agoDiscussion last month when the preprint was released: https://news.ycombinator.com/item?id=33823170 https://news.ycombinator.com/item?id=33823170
- deleted 4y ago[deleted]
- Kalanos 4y agoWhat exactly is the negative data? Seems like it's just scrambled nonsense (aka what the truth is not)
- abraxas 4y agoI think it is scrambled nonsense but it's scrambled in a way that still makes it look like a plausible sample. I remember watching a video of Hinton saying that just using white noise or similarly randomized data does not work but I'm now forgetting why.
- ipnon 4y agoIt’s incredible to think that dreams are just our brains generating training data, and lack of sleep causes us to overfit on our immediate surroundings.
- mlajtos 4y agoWe tend to start hallucinating when we don't have enough sleep. So generating training data is necessary, but way safer when our muscles are turned off.
- mherdeg 4y agoThanks for this little comment thread folks! This makes me cheerful because it suggests a way that studying systems which appear intelligent might be able to teach us more about how human intelligence works.
- snowpid 4y agoin addition: During pre electricity time humen woke up after 4 hours sleep, got awake for some time and then continue to sleep. My guess, this sleep pattern is better for learning.
- davidgay 4y ago> During pre electricity time humen woke up after 4 hours sleep, got awake for some time and then continue to sleep. The confusing thing with this claim is what did people actually do during this time, given bad (and expensive!) lighting only?
- snowpid 4y agoThinking, talking to peers, Smartphones are a new invention and night was a dangerous time for a long time.
- mlajtos 4y agoIt's called biphasic sleep for people that want to read up on it. > My guess, this sleep pattern is better for learning. That might be true. One of the techniques to induce lucid dreaming works similarly – sleep for 4-5 hours, wake up, stay awake for 15-60mins then go back to sleep. It's called "wake back to bed" technique. Many lucid dreamers report increased capacity for learning in dreams.
- singularity2001 4y agoThe paragraph about Mortal Computation is worth repeating: If these FF networks can be proven to scale or made to scale similarly to BP networks, this would enable making hardware several orders of magnitude more efficient, for the price of loosing the ability to make exact copies of models to other computers. (The loss of reproducibility sits well with the tradition of scientific papers anyway/s;) 2.) How does this paper relate to Hintons feedback alignment from 5 years ago? I remember it was feedback without derivatives. What are the key new ideas? To adjust the output of each individual layer to be big for positive cases and small for negative cases without any feedback? Have these approaches been combined?
- parpfish 4y agoHaving read through the forward-forward paper, it feels like it's Oja's rule adapted for supervised learning but I can't really articulate why...
- jasonjmcghee 4y agoMaybe I'm missing something, but from the paper https://www.cs.toronto.edu/~hinton/FFA13.pdf https://www.cs.toronto.edu/~hinton/FFA13.pdf, they use non-conv nets on CIFAR-10 for back prop, resulting in 63% accuracy. And FF achieves 59% accuracy (at best). Those are relatively close figures, but good accuracy on CIFAR-10 is 99%+ and getting ~94% is trivial. So, if an improper architecture for a problem is used and the accuracy is poor, how compelling is using another optimization approach and achieving similar accuracy? It's a unique and interesting approach, but the article specifically mentions it gets accuracy similar to backprop, but if this is the experiment that claim is based on, it loses some credibility in my eyes.
- whimsicalism 4y agoYou have to start with toy models before scaling up.
- electrograv 4y agoAchieving <80% on CIFAR10 in the year >2020 is an example of a failed toy model, not a successful toy model. Almost any ML algorithm can be thrown at CIFAR10 and achieve ~60% accuracy; this ballpark of accuracy is really not sufficient to demonstrate viability, no matter how aesthetically interesting the approach might feel.
- deleted 4y ago[deleted]
- alper111 4y agoPlain MLP acc. is 63% vs 59% with FF, not so bad? By the same logic, MLP is a failed toy model.
- jasonjmcghee 4y agoI haven't seen this to be the case, fwiw. There was a paper in 2016 that did this and most were in the ~40% range. But "any ml algorithm" isn't the point. It's a new optimization technique and should be applied to models/architectures that make sense with the problems they are being used on. For example, they could have used a pretrained featurizer and trained the two layer model on top of it, with both back prop and FF and compared.
- cschmid 4y agoThe article links to an old draft of the paper (it seems that the results in 4.1 couldn't be replicated). The arxiv has a more recent one: https://arxiv.org/abs/2212.13345 https://arxiv.org/abs/2212.13345
- belter 4y ago@dang Meta question on HN implementation: Why do sometimes submitting a previously submitted resource links automatically to the previous discussion, while other times is considered a new submission?
- gregschlom 4y agoAs far as I know it's a simple string match on the url. If the url is different (for example a new anchor tag is added) then it's considered a new submission.
- danuker 4y agoIf you click on "past" under this submission, you see two identical URLs: https://hn.algolia.com/?query=Geoffrey%20Hinton%20publishes%20new%20deep%20learning%20algorithm&type=story&dateRange=all&sort=byDate&storyText=false&prefix&page=0 https://hn.algolia.com/?query=Geoffrey%20Hinton%20publishes%...
- mdp2021 4y agoWhich is odd, because I checked in the minutes following the submission and I remember "[past]" did not return anything.
- drdeca 4y agoI believe one factor is the amount of time between the submissions.
- sva_ 4y agoI found this paragraph from the paper very interesting: > 7 The relevance of FF to analog hardware > An energy efficient way to multiply an activity vector by a weight matrix is to implement activities as voltages and weights as conductances. Their products, per unit time, are charges which add themselves. This seems a lot more sensible than driving transistors at high power to model the individual bits in the digital representation of a number and then performing O(n^2) single bit operations to multiply two n-bit numbers together. Unfortunately, it is difficult to implement the backpropagation procedure in an equally efficient way, so people have resorted to using A-to-D converters and digital computations for computing gradients (Kendall et al., 2020). The use of two forward passes instead of a forward and a backward pass should make these A-to-D converters unnecessary. It was my impression that it is difficult to properly isolate an electronic system to use voltages in this way (hence computers sort of "cut" voltages into bits 0/1 using a step function). Have these limitations been overcome or do they not matter as much, as neural networks can work with more fuzzy data? Interesting to imagine such a processor though.
- btown 4y agoPhotonic/optical neural networks are an interesting related area of research, using light interference to implement convolution and other operations without (I believe?) needing a bitwise representation of intensity. https://www.nature.com/articles/s41467-020-20719-7 https://www.nature.com/articles/s41467-020-20719-7 https://opg.optica.org/optica/fulltext.cfm?uri=optica-5-7-864&id=395466 https://opg.optica.org/optica/fulltext.cfm?uri=optica-5-7-86...
- CuriouslyC 4y agoThe small deltas resulting from electrical noise generally aren't an issue for probabilistic computations. Interestingly, people have quantized many large DL models down to 8/16 bits, and accuracy reduction is often on the order of 2-5%. Additionally, adding random noise to weights during training tends to act as a form of regularization.
- Animats 4y agoThere's been unhappiness in some quarters that back propagation doesn't seem to be something that biology does. That may be part of the motivation here.
- Coneylake 4y agoIs the derivative calculated by forward-forward as analytic as in backpropagation?
- keepquestioning 4y agoIs this a game changer?
- penciltwirler 4y agoThis is old news already.
- NHQ 4y agoHinton's networks become the neuron of novel networks. It is important to know that these types of weights don't learn features, they map a compressed representation of the learned info, which is the input. Classification through error correction. That is actually what labels do for supervised learning (IOW they learn many ways to represent the label, and that is what the weights are). Modern AI do that plus learn features, but the weights are nevertheless a representation of what was learned, plus a fancy way to encode and decode into that domain. What Hinton and Deepmind will do is use neural-network learned-data, or perhaps the weights, as input to this kind of network. In other words, the output of another NN is labeled a priori, ergo you can use it "unsupervised" networks, which this research expounds. This will allow them to cook the input network into a specific dish, by labels even. Now give me my phd. edit: edit
- emilecourth 4y agoThere's an open source implementation of the paper in pytorch https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/forward_forward https://github.com/nebuly-ai/nebullvm/tree/main/apps/acceler... by @diegofiori_ He also wrote an interesting thread on the memory usage of this algo versus backprop https://twitter.com/diegofiori_/status/1605242573311709184?s=20&t=WjJVGJ_VOFuUKpRigaLLRw https://twitter.com/diegofiori_/status/1605242573311709184?s...