10 ms·

I'm one of the speakers at the workshop mentioned [1]. The article is a bit of a concept salad. I'm not familiar with all of the papers mentioned but am happy

I'm one of the speakers at the workshop mentioned [1]. The article is a bit of a concept salad. I'm not familiar with all of the papers mentioned but am happy to try to answer questions. [1] https://beyondbackprop.github.io/ https://beyondbackprop.github.io/

pretty_dumm_guy 6y ago
Hi Professor, Good day, I was wondering whether it is be possible for you to provide an overview of different methods that you think might have a better shot at replacing backpropagation algorithm?
- duvenaud 6y ago
  Sure. First of all, I want to say that backprop, by which I mean reverse-mode differentiation for computing gradients, combined with gradient descent for updating parameters, is pretty great. In a sense it's the last thing we should be trying to replace, since pretty much the whole deep learning revolution was about replacing hand-designed functions with ones that can be optimized in this way. Reverse-mode differentiation has about the same time cost as whatever function you're optimizing, no matter how many parameters you need gradients for. This which is about as good as one could hope for, and is what lets it scale to billions of parameters. The main downside of reverse-mode differentiation (and one of the reasons it's biologically implausible) is that it requires storing all the intermediate numbers that were computed when evaluating the function on the forward pass. So its memory cost grows with the complexity of the function being optimized. So the main practical problem with reverse-mode differentiation + gradient descent is the memory requirement, and much of the research presented in the workshop is about ways to get around this. A few of the major approaches are: 1) Only storing a subset of the forward activations, to get noisier gradients at less memory cost. This is what the "Randomized Automatic Differentiation" paper does. You can also save memory and get exact gradients if you re-construct the activations as you need them (called checkpointing), but this is slower. 2) Only training one layer at a time. This is what the "Layer-wise Learning" papers are doing. I suppose you could also say that this is what the "feedback alignment" papers are doing. 3) If the function being optimized is a fixed-point computation (such as an optimization), you can compute its gradient without needing to store any activations by using the implicit function theorem. This is what my talk was about. 4) Some other forms of sensitivity analysis (not exactly the same as computing gradients) can be done by just letting a dynamical system run for a little while. Barak Pearlmutter has some work on how he thinks this is what happens in slow-wave sleep to make our brains less prone to seizures when we're awake. I'm missing a lot of relevant work, and again I don't even know all the work that was presented at this one workshop. But I hope this helps.
  - pretty_dumm_guy 6y ago
    Thank you for your answer. It appears to me that we are trying to achieve an algorithm that has better time complexity than the one that we have right now(reverse mode differentiation with gradient descent). Is it possible to combine these methods in a straight forward manner with methods that try to reduce the space complexity? For example, Lottery ticket hypothesis(https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635) seems to reduce spacial complexity(Please do correct me if I am wrong). Also, based on my rather poor and limited knowledge, it appears to me that set of proposed methods that reduced space complexity and set of proposed methods that reduce time complexity are disjoint. Is that the case ?
    - sdenton4 6y ago
      (Lottery Ticket, to date, produces small networks ex post facto... You still have to train the giant network. There's also some indication that it's chancy on 'large' datasets+problems. https://arxiv.org/abs/1902.09574 https://arxiv.org/abs/1902.09574 )
    - duvenaud 6y ago
      Thanks for your question! But as I said, no one is really worried about the asymptotic time complexity of reverse mode differentiation, although there is scope for improving constants). The main scope for improvement is in the space complexity. There is a lot of work on trying to speed up optimization, for example the K-FAC optimizer by Roger Grosse that uses second-order gradient information in a scalable way. The lottery ticket pruning strategies do reduce space complexity, but I think the main reason people are interested in it is to reduce training time complexity, or deployment memory requirements, but not so much training memory requirements. As for whether memory-saving and time-saving approaches are disjoint, many methods (like checkpointing) introduce a tradeoff between time and space complexity, so no.
      - pretty_dumm_guy 6y ago
        Thank you again for the clarifications. You have given me something to chew on over the holidays. I wish you and your family a happy Christmas :)
im3w1l 6y ago
One approach I've been thinking about is optimizing each neuron using only the global loss and information about the neighboring neurons. Basically if the network made the correct prediction tell each neuron to do a little bit more of what it just did. If it sent a high output, change the weights so it sends an even higher output. Weaken connections that were inhibitory and strengthen connections that were excitatory. And for a neuron with a low output, make it even lower by doing the opposite. If on the other hand prediction was wrong, then try to make the neuron do less of what it did. Do you know if something like this has been tried?
- sdenton4 6y ago
  Backprop is good at giving credit where credit is due: you're looking at the impact of each weight on loss, which allows changing each weight to improve the loss, by an appropriate amount proportional to the other weights. You can even have some negative weight gradients and some positive; ie, it may be that even with a 'good' overall result that it's best to turn down a particular weight. So my guess is that this approach would either take a much longer time to converge (as there's less information transmitted back for the neuron updates) or stall out completely. Probably not too hard to code up, if you want to try it. But I would also be pretty surprised if it hadn't been tried before.
  - im3w1l 6y ago
    I agree it's likely to converge slower, but I was thinking that it could maybe make up for it by being cheaper. Based on what I know it also seems more biologically plausible that learning is a mostly local process.
- duvenaud 6y ago
  This is basically what the chain rule / backprop does. The only caveat is that it's pretty hard to make a network with only positive weights. And once you allow negative weights, you have to know whether your downstream neighbours should have had higher or lower output. The easiest way to do this is to propagate the gradient backwards from the output, hence backprop.
interblag 6y ago
Thanks for offering your insight! Slightly tangential question, but on reading the article I was surprised that it mentioned multiple anonymous submissions, given that this was a workshop at one of the most prestigious ML conferences. Any particular reason for this that you can think of?
- duvenaud 6y ago
  The names of all the authors are clearly listed on the workshop website [1]. I think this is just a case of incompetence on the part of the journalist. https://beyondbackprop.github.io/ https://beyondbackprop.github.io/
  - Ar-Curunir 6y ago
    The submission process is anonymous, but after papers are accepted there’s no need for anonymity, so author names are published. Don’t see why youre questioning the competence of the journalist if you don’t know the submission process yourself.
    - duvenaud 6y ago
      To be charitable, you could call it misleading laziness. But the article was published long after all the papers were de-anonymized, so it seems perverse to explain that the submissions were originally anonymous instead of just giving the authors' names. Furthermore, the journalist managed to write an entire article about a workshop without once naming it, giving a link, or even defining backpropagation. Edit: I want to say that I understand it's hard to cover an area that you're not an expert in. But if the journalist had googled the title of the articles he was writing about, he would have found the authors' names. Instead he gave the reader the impression that the articles are still anonymous.
- Ar-Curunir 6y ago
  Submissions are anonymous, but after acceptance papers are unblinded
master_yoda_1 6y ago
Are you not worried that this article has a lot of misinformation and these kind of workshop are very common in NIPS. In 2015 there is a "quantum machine learning" workshop https://nips.cc/Conferences/2015/ScheduleMultitrack?event=4934 https://nips.cc/Conferences/2015/ScheduleMultitrack?event=49... I can easily write an article, "deep learning is dead, here comes quantum ml" How you academic stop this bullshit, or are you yourself a bullshit who spread misinformation by supporting these kind of articles?
- duvenaud 6y ago
  Good question. I do think there are some bogus workshops, and at least a couple of the papers at this workshop seem to be written by people who don't understand the area very well. However, there's also a lot of great work in this workshop. It's also asking an important and fundamental question, which we have reason to believe has better answers than our current ones. I also think quantum ML is a perfectly sensible line of inquiry (although imo unlikely to pay off for a long time, if ever), even if its results are consistently overhyped in the media. I don't really know what to do about the state of journalism, however. I thought I was being pretty hard on the author of this piece elsewhere in this thread, actually. I guess you could blame everyone who upvoted the article. On the other hand, we can't let the perfect be the enemy of the good. There is a screening process for NIPS workshops (which I've been part of the last two years) where more experienced researchers rate the proposed workshops. There are usually a couple that get through that I think shouldn't be. However, I'd rather let 1000 stupid workshops through than shut down the one seemingly weird one that's actually a promising direction. Because if not here, where can we nurture and support weird ideas? Finally, I think workshops play a vital role in onboarding newcomers to the field. One of the main ways you learn is by writing papers, and workshops lower the barrier to getting something out the door and getting feedback on it.
  - master_yoda_1 6y ago
    Workshop is not bogus the way you are commercializing is bogus. Research is incremental process and there are lot of frontier scientist currently working on. The importance given to only this workshop is bullshit

pretty_dumm_guy 6y ago

Hi Professor, Good day, I was wondering whether it is be possible for you to provide an overview of different methods that you think might have a better shot at replacing backpropagation algorithm?

duvenaud 6y ago
Sure. First of all, I want to say that backprop, by which I mean reverse-mode differentiation for computing gradients, combined with gradient descent for updating parameters, is pretty great. In a sense it's the last thing we should be trying to replace, since pretty much the whole deep learning revolution was about replacing hand-designed functions with ones that can be optimized in this way. Reverse-mode differentiation has about the same time cost as whatever function you're optimizing, no matter how many parameters you need gradients for. This which is about as good as one could hope for, and is what lets it scale to billions of parameters. The main downside of reverse-mode differentiation (and one of the reasons it's biologically implausible) is that it requires storing all the intermediate numbers that were computed when evaluating the function on the forward pass. So its memory cost grows with the complexity of the function being optimized. So the main practical problem with reverse-mode differentiation + gradient descent is the memory requirement, and much of the research presented in the workshop is about ways to get around this. A few of the major approaches are: 1) Only storing a subset of the forward activations, to get noisier gradients at less memory cost. This is what the "Randomized Automatic Differentiation" paper does. You can also save memory and get exact gradients if you re-construct the activations as you need them (called checkpointing), but this is slower. 2) Only training one layer at a time. This is what the "Layer-wise Learning" papers are doing. I suppose you could also say that this is what the "feedback alignment" papers are doing. 3) If the function being optimized is a fixed-point computation (such as an optimization), you can compute its gradient without needing to store any activations by using the implicit function theorem. This is what my talk was about. 4) Some other forms of sensitivity analysis (not exactly the same as computing gradients) can be done by just letting a dynamical system run for a little while. Barak Pearlmutter has some work on how he thinks this is what happens in slow-wave sleep to make our brains less prone to seizures when we're awake. I'm missing a lot of relevant work, and again I don't even know all the work that was presented at this one workshop. But I hope this helps.
- pretty_dumm_guy 6y ago
  Thank you for your answer. It appears to me that we are trying to achieve an algorithm that has better time complexity than the one that we have right now(reverse mode differentiation with gradient descent). Is it possible to combine these methods in a straight forward manner with methods that try to reduce the space complexity? For example, Lottery ticket hypothesis(https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635) seems to reduce spacial complexity(Please do correct me if I am wrong). Also, based on my rather poor and limited knowledge, it appears to me that set of proposed methods that reduced space complexity and set of proposed methods that reduce time complexity are disjoint. Is that the case ?
  - sdenton4 6y ago
    (Lottery Ticket, to date, produces small networks ex post facto... You still have to train the giant network. There's also some indication that it's chancy on 'large' datasets+problems. https://arxiv.org/abs/1902.09574 https://arxiv.org/abs/1902.09574 )
  - duvenaud 6y ago
    Thanks for your question! But as I said, no one is really worried about the asymptotic time complexity of reverse mode differentiation, although there is scope for improving constants). The main scope for improvement is in the space complexity. There is a lot of work on trying to speed up optimization, for example the K-FAC optimizer by Roger Grosse that uses second-order gradient information in a scalable way. The lottery ticket pruning strategies do reduce space complexity, but I think the main reason people are interested in it is to reduce training time complexity, or deployment memory requirements, but not so much training memory requirements. As for whether memory-saving and time-saving approaches are disjoint, many methods (like checkpointing) introduce a tradeoff between time and space complexity, so no.
    - pretty_dumm_guy 6y ago
      Thank you again for the clarifications. You have given me something to chew on over the holidays. I wish you and your family a happy Christmas :)

im3w1l 6y ago

One approach I've been thinking about is optimizing each neuron using only the global loss and information about the neighboring neurons. Basically if the network made the correct prediction tell each neuron to do a little bit more of what it just did. If it sent a high output, change the weights so it sends an even higher output. Weaken connections that were inhibitory and strengthen connections that were excitatory. And for a neuron with a low output, make it even lower by doing the opposite. If on the other hand prediction was wrong, then try to make the neuron do less of what it did. Do you know if something like this has been tried?

sdenton4 6y ago
Backprop is good at giving credit where credit is due: you're looking at the impact of each weight on loss, which allows changing each weight to improve the loss, by an appropriate amount proportional to the other weights. You can even have some negative weight gradients and some positive; ie, it may be that even with a 'good' overall result that it's best to turn down a particular weight. So my guess is that this approach would either take a much longer time to converge (as there's less information transmitted back for the neuron updates) or stall out completely. Probably not too hard to code up, if you want to try it. But I would also be pretty surprised if it hadn't been tried before.
- im3w1l 6y ago
  I agree it's likely to converge slower, but I was thinking that it could maybe make up for it by being cheaper. Based on what I know it also seems more biologically plausible that learning is a mostly local process.
duvenaud 6y ago
This is basically what the chain rule / backprop does. The only caveat is that it's pretty hard to make a network with only positive weights. And once you allow negative weights, you have to know whether your downstream neighbours should have had higher or lower output. The easiest way to do this is to propagate the gradient backwards from the output, hence backprop.

interblag 6y ago

Thanks for offering your insight! Slightly tangential question, but on reading the article I was surprised that it mentioned multiple anonymous submissions, given that this was a workshop at one of the most prestigious ML conferences. Any particular reason for this that you can think of?

duvenaud 6y ago
The names of all the authors are clearly listed on the workshop website [1]. I think this is just a case of incompetence on the part of the journalist. https://beyondbackprop.github.io/ https://beyondbackprop.github.io/
- Ar-Curunir 6y ago
  The submission process is anonymous, but after papers are accepted there’s no need for anonymity, so author names are published. Don’t see why youre questioning the competence of the journalist if you don’t know the submission process yourself.
  - duvenaud 6y ago
    To be charitable, you could call it misleading laziness. But the article was published long after all the papers were de-anonymized, so it seems perverse to explain that the submissions were originally anonymous instead of just giving the authors' names. Furthermore, the journalist managed to write an entire article about a workshop without once naming it, giving a link, or even defining backpropagation. Edit: I want to say that I understand it's hard to cover an area that you're not an expert in. But if the journalist had googled the title of the articles he was writing about, he would have found the authors' names. Instead he gave the reader the impression that the articles are still anonymous.
Ar-Curunir 6y ago
Submissions are anonymous, but after acceptance papers are unblinded

master_yoda_1 6y ago

Are you not worried that this article has a lot of misinformation and these kind of workshop are very common in NIPS. In 2015 there is a "quantum machine learning" workshop https://nips.cc/Conferences/2015/ScheduleMultitrack?event=4934 https://nips.cc/Conferences/2015/ScheduleMultitrack?event=49... I can easily write an article, "deep learning is dead, here comes quantum ml" How you academic stop this bullshit, or are you yourself a bullshit who spread misinformation by supporting these kind of articles?

duvenaud 6y ago
Good question. I do think there are some bogus workshops, and at least a couple of the papers at this workshop seem to be written by people who don't understand the area very well. However, there's also a lot of great work in this workshop. It's also asking an important and fundamental question, which we have reason to believe has better answers than our current ones. I also think quantum ML is a perfectly sensible line of inquiry (although imo unlikely to pay off for a long time, if ever), even if its results are consistently overhyped in the media. I don't really know what to do about the state of journalism, however. I thought I was being pretty hard on the author of this piece elsewhere in this thread, actually. I guess you could blame everyone who upvoted the article. On the other hand, we can't let the perfect be the enemy of the good. There is a screening process for NIPS workshops (which I've been part of the last two years) where more experienced researchers rate the proposed workshops. There are usually a couple that get through that I think shouldn't be. However, I'd rather let 1000 stupid workshops through than shut down the one seemingly weird one that's actually a promising direction. Because if not here, where can we nurture and support weird ideas? Finally, I think workshops play a vital role in onboarding newcomers to the field. One of the main ways you learn is by writing papers, and workshops lower the barrier to getting something out the door and getting feedback on it.
- master_yoda_1 6y ago
  Workshop is not bogus the way you are commercializing is bogus. Research is incremental process and there are lot of frontier scientist currently working on. The importance given to only this workshop is bullshit