9 ms·
Researchers propose faster, more efficient alternatives to backpropagation
- webmaven 6y agoAside from being an excellent overview of NeurIPS 2020 papers on this topic, I found it curious that several of them were anonymous. Are anonymously submitted papers becoming (more) common? If so, what's driving this?
- gwern 6y agoI'd assume they just haven't been unblinded yet.
- webmaven 6y agoHmm. Shouldn't papers all be unblinded at once when acceptances/rejections are sent out?
- dkislyuk 6y agoThese are workshop submissions (which typically implies a more lightweight review process, for more exploratory work), and it is possible the same submissions are currently in blind review for other conferences in their final form.
- webmaven 6y ago> [I]t is possible the same submissions are currently in blind review for other conferences in their final form. Ah, that makes sense.
- duvenaud 6y agoThey were all unblinded weeks ago, the authors are listed on the workshop website: https://beyondbackprop.github.io/ https://beyondbackprop.github.io/
- webmaven 6y agoSo, at this point, the most likely explanation seems to be that TFA was written prior to the unblinding, and then wasn't subsequently reviewed or updated before publication.
- cl3misch 6y agoI find the nomenclature in this article a bit weird. > Another disadvantage of backpropagation is its tendency to become stuck in the local minima of the loss function. Mathematically, the goal in training a model is converging on the global minimum, the point in the loss function where the model has optimized its ability to make predictions. "Backpropagation" is the method how to compute the gradient of the weights with respect to a loss function. But the article repeatedly uses the term as if it was the whole optimization algorithm, running into local minima.
- savant_penguin 6y agoSome people use backpropagation and gradient decent interchangeably. It's really confusing though
- jhrmnn 6y agoI wouldn’t say it’s confusing—it’s wrong and suggests at the least sloppiness in terminology
- ethbr0 6y ago> sloppiness in terminology venturebeat.com Not the worst, but we're not talking Nature or Spectrum here.
- option 6y agoof course it is wrong. One nice thing about math is that thing are defined precisely and back propagation and, say, SGD or Adam are different things
- dnautics 6y agoThat's bad. There are gradient descent techniques that are feed-forward, and understand the domains where those are appropriate and the domains where backprop is appropriate is I think important, especially as you try to do things like mix machine learning techniques with other differentiable programming strategies.
- 6y ago
- dr_j_ 6y agoSo assign random values to connection weights and then ‘spin’ those weights to a combination of other random values that hopefully perform a bit more favourably.. isn’t this just random search?
- quotemstr 6y agoIt's not a random search through the parameter space: "But how do we select a good network from these Kn different networks? Brute-force evaluation of all possible configurations is clearly not feasible due to the massive number of different hypotheses. Instead, we present an algorithm, shown in Figure 1, that iteratively searches the best combination of connection values for the entire network by optimizing the given loss. To do this, the method learns a real-valued quality score for each weight option. These scores are used to select the weight value of each connection during the forward pass. The scores are then updated in the backward pass based on the loss value in order to improve training performance over iterations." It's actually pretty clever.
- sdenton4 6y agoIt reminds me of Bayesian model sampling, where you have a distribution over possible weights and 'draw' a model from the distribution for each evaluation... A problem is that there may be interesting co-dependencies amongst the weights which independent sampling will have a hard time getting right.
- ssivark 6y agoRandom search is a technical term in optimization with a very specific meaning (which unfortunately does not mean searching random locations in parameter space a la brute force). It’s more in the spirit of randomly deciding the direction in which to try to take the next step, thereby implicitly deriving a gradient component by sampling. https://en.m.wikipedia.org/wiki/Random_search https://en.m.wikipedia.org/wiki/Random_search
- deleted 6y ago[deleted]
- sesuximo 6y agoCan someone ELI am an undergrad? I don’t see how gradient descent “forgets” anything
- saiojd 6y agoGradient decent doesn't per se, but retraining ("fine-tuning") on another dataset forgets most of the training done on the first dataset.
- joshgel 6y agoIs this true? My understanding was that in fine tuning, you’d only re train some of the layers. And even if you re train all the layers, the starting point for the layers is not random. If it really was all forgotten then fine tuning would not be orders of magnitude faster...
- thunderbird120 6y agoGradient decent optimizes performance of a model on a given dataset. If you stop training on one dataset and start training on another one your model will become more optimized for the second dataset and less optimized for the first. This will usually result in degraded performance on classes of data found more commonly in the first dataset but not the second. This is what people mean by "forgetting". It doesn't matter how much of the model you fine-tune, the effect is still present though the effect size varies.
- unishark 6y agoIts not a complicated concept, just a stretch of the concept of memory. Training in deep learning is done in batches. So "learning" (i.e. the gradient updates to weights) that happens due to your early batches of data can be undone by the gradient updates for later batches. The gradient in machine learning is based on the loss. Specifically it's the direction that reduces the loss the fastest. So, not only the most recent batches, but specifically by the recent data that is predicted incorrectly. It doesn't have any "confidence" from the memory of what was predicted right previously, for example, it just currently only cares about changing to suit the most recent batches.
- mlthoughts2018 6y agoI found it very weird that the SLIDE algorithm from early 2019 isn’t mentioned. Maybe I missed it or maybe it is compared just deeper in the referenced publications? SLIDE seems way, way superior to any of the listed solutions or approaches, as far as I could tell on a first read through. https://arxiv.org/abs/1903.03129 https://arxiv.org/abs/1903.03129
- quotemstr 6y agoAIUI, that only works on sparse networks
- mlthoughts2018 6y agoBut there’s also been a lot of research suggesting most SOTA dense networks are arbitrarily replicatable with sparse networks, and may even be better in the sense of less overfitting. Perhaps things like GPT are still an exception, but for most applications SLIDE should work to train networks just as effective as naively specified dense architectures.
- quotemstr 6y agoYeah. I think part of the problem is just that SLIDE represents a Kuhnesque paradigm shift and these things take time. I really want to play with SLIDE myself but just haven't had a chance.
- cinquemb 6y ago> But there’s also been a lot of research suggesting most SOTA dense networks are arbitrarily replicatable with sparse networks I'm not sure if its related, but would this work kind of how armadillo can do singular value decomp [0] of a matrix by embedding arbitrary n by m matrix X in a higher dimensional n+m by n+m null matrix M? [0] http://arma.sourceforge.net/docs.html#svds http://arma.sourceforge.net/docs.html#svds
- manjunaths 6y agohttps://beyondbackprop.github.io/ https://beyondbackprop.github.io/ This is the NeurIPS workshop that the article is talking about.
- bra-ket 6y agoThe best part of the article is a quote by G. Hinton, the father of "deep learning", at the end: "My view is throw it all away and start again, I don’t think that’s how the brain works."
- officehero 6y agoThe main problem is not backpropagation though, but the fixation of resources on DL projects (that's what I call local minimum!). In my department, for example, they don't seem to care about the application, integration, deployment etc, as long as it's DL or DRL.
- faitswulff 6y agoDidn’t he recently publish an article about a drastically different way to approach machine learning called capsule networks?
- p1esk 6y agoThere's nothing "drastically different" about capsules networks.
- deehouie 6y agoWhile Hinton's view need to be noted, I heard a quote attributed to Yann LeCun, something like, "If you want to learn flying by modeling the biology of birds, you're doing it wrong. Just look at today's airplanes. They have no resemblance to birds at all. Yet they're million times better and faster than any bird."
- bra-ket 6y agothe problem with that analogy is trying to build an airplane before you figured out the laws of physics.
- hyko 6y agoPeople build things without understanding the underlying principles all the time, e.g. the steam engine. You could probably make the case that building things has helped our understanding more than our understanding has helped us build things. Having said that, you can certainly improve a design when you better understand the fundamentals (vs intuition + trial & error).
- thweroiuorier 6y agoMeh. Seems like a lot of hot air over nothing significant.
- duvenaud 6y agoI'm one of the speakers at the workshop mentioned [1]. The article is a bit of a concept salad. I'm not familiar with all of the papers mentioned but am happy to try to answer questions. [1] https://beyondbackprop.github.io/ https://beyondbackprop.github.io/
- pretty_dumm_guy 6y agoHi Professor, Good day, I was wondering whether it is be possible for you to provide an overview of different methods that you think might have a better shot at replacing backpropagation algorithm?
- duvenaud 6y agoSure. First of all, I want to say that backprop, by which I mean reverse-mode differentiation for computing gradients, combined with gradient descent for updating parameters, is pretty great. In a sense it's the last thing we should be trying to replace, since pretty much the whole deep learning revolution was about replacing hand-designed functions with ones that can be optimized in this way. Reverse-mode differentiation has about the same time cost as whatever function you're optimizing, no matter how many parameters you need gradients for. This which is about as good as one could hope for, and is what lets it scale to billions of parameters. The main downside of reverse-mode differentiation (and one of the reasons it's biologically implausible) is that it requires storing all the intermediate numbers that were computed when evaluating the function on the forward pass. So its memory cost grows with the complexity of the function being optimized. So the main practical problem with reverse-mode differentiation + gradient descent is the memory requirement, and much of the research presented in the workshop is about ways to get around this. A few of the major approaches are: 1) Only storing a subset of the forward activations, to get noisier gradients at less memory cost. This is what the "Randomized Automatic Differentiation" paper does. You can also save memory and get exact gradients if you re-construct the activations as you need them (called checkpointing), but this is slower. 2) Only training one layer at a time. This is what the "Layer-wise Learning" papers are doing. I suppose you could also say that this is what the "feedback alignment" papers are doing. 3) If the function being optimized is a fixed-point computation (such as an optimization), you can compute its gradient without needing to store any activations by using the implicit function theorem. This is what my talk was about. 4) Some other forms of sensitivity analysis (not exactly the same as computing gradients) can be done by just letting a dynamical system run for a little while. Barak Pearlmutter has some work on how he thinks this is what happens in slow-wave sleep to make our brains less prone to seizures when we're awake. I'm missing a lot of relevant work, and again I don't even know all the work that was presented at this one workshop. But I hope this helps.
- zipotm 6y agoBullshit, backpropagation was discovered by Rosenblatt...
- bra-ket 6y agoI'd say Gottfried W. Leibniz is the true author, as it's all comes down to calculus. The particular implementation for "neural nets" is just a special case of function minimization by taking derivatives.
- contingencies 6y agoI like zoom-out views. To push what you describe further, it is essentially what ancient humans or their non-hominid forebears did subconsciously when calculating optimum motion trajectories to catch or spear prey while hunting... merely a version in formal notation ... we can thank the zero of India (https://en.wikipedia.org/wiki/0#History https://en.wikipedia.org/wiki/0#History), the Persians (https://en.wikipedia.org/wiki/Algorithm#Etymology https://en.wikipedia.org/wiki/Algorithm#Etymology), the Islamic renaissance in Europe (https://mitpress.mit.edu/books/islamic-science-and-making-european-renaissance https://mitpress.mit.edu/books/islamic-science-and-making-eu...) and numerous others for the slow development of the requisite formal maths. But a rose by any other name would smell as sweet. And perhaps, in the context of the stupefyingly deferred emergence of zero, even nameless!
- webmaven 6y ago> To push what you describe further, it is essentially what ancient humans or their non-hominid forebears did subconsciously when calculating optimum motion trajectories to catch or spear prey while hunting Hah! So, invention recapitulates evolution?
- contingencies 6y agoZero to one!
- lock-free 6y ago
- chronolitus 6y agoAn interesting biologically-inspired alternative to backpropagation is STDP (https://en.wikipedia.org/wiki/Spike-timing-dependent_plasticity https://en.wikipedia.org/wiki/Spike-timing-dependent_plastic...) However, it's still quite hard to get useful results from it in practice. --------------- I personally believe that one component of intelligence is the ability to apply cognitive patterns created for a particular input to other inputs. (Very simplified example: A block of "neurons" which have learned to recognize the pattern "is hurt by" when given a subject (group of pixels in image) and object (other group of pixels in image), could be applied to another subject/object pair, for example coming from processed audio. But if the audio processing takes 10 layers, and the image processing 5, the connection has to run backwards) To do this in a state of the art deep network, you need the ability to create backward connections. Backward connections imply loops, and loops break backprop (unlike loops in RNNs, which can be easily unrolled AFAIK). So with the current backprop-trained feedforward model, you have to create patterns multiple time instead of reusing them. This is why I will pay attention to backprop alternatives which allow loops, despite their (currently many) disadvantages. This and modular training are the two aspects of learning I would personally focus on.
- alok-g 6y agoCan you pls. point to relevant literature showing attempts on using STDP. I am not exactly working on this, however, intuitively believe that it should be possible and effective to make STDP work (including handling loops). Thanks.