4 ms·

Direct link to the Google Brain paper, which is well worth a read: https://dl.acm.org/citation.cfm?id=3321441 https://dl.acm.org/citation.cfm?id=3321441 (click

by cs702 7y ago

Direct link to the Google Brain paper, which is well worth a read: https://dl.acm.org/citation.cfm?id=3321441 https://dl.acm.org/citation.cfm?id=3321441 (click on "PDF" link to read) Abstract: "In this paper we argue that systems for numerical computing are stuck in a local basin of performance and programmability. Systems researchers are doing an excellent job improving the performance of 5-year-old benchmarks, but gradually making it harder to explore innovative machine learning research ideas. We explain how the evolution of hardware accelerators favors compiler back ends that hyper-optimize large monolithic kernels, show how this reliance on high-performance but inflexible kernels reinforces the dominant style of programming model, and argue these programming abstractions lack expressiveness, maintainability, and modularity; all of which hinders research progress. We conclude by noting promising directions in the field, and advocate steps to advance progress towards high-performance general purpose numerical computing systems on modern accelerators." The main example the authors use to illustrate these issues is capsule networks, first proposed two years ago.[a] To date, no one has been able to develop a high-performance implementation of capsule networks. At present, the best-performing implementations in Tensorflow and PyTorch must copy, rearrange, and materialize to memory two orders of magnitude more data than necessary, due to the issues raised by the authors. See sections 1 and 2 of the paper for the gory details. Two orders of magnitude. That is pathetic. [a] https://arxiv.org/abs/1710.09829 https://arxiv.org/abs/1710.09829

cr0sh 7y ago
Note: I am not an expert on ML, ANNs, etc - most of what I have done has been mainly "hobby level" and MOOC learning. From what I understand, though, what GH is trying to accomplish with capsule networks (which I have tried to understand, and have yet to succeed) is optimizing (or possible removal of) backpropagation. He has noted in the past how - so far (from what I know) - there isn't a biological equivalent to backpropagation for learning. Backprop is a completely artificial mathematical construct that doesn't happen in natural systems. It is also extremely energy intensive - at least in the manner that it is currently done. So the question is - that he's hoping to answer I think - what is a proper working framework for implementing an artificial neural network that can learn, without using backpropagation (or using it differently)? I think whoever can solve this will fundamentally remake the field of ML/AI - much in the same way that backprop (and later "deep learning") did.
- visarga 7y ago
  It's not the backprop that is holding back ML - it takes about as much time as the forward pass and requires 3x the memory. Backprop is necessary for almost all the deep neural nets that have state of the art results, attempting to replace it would push us back a decade or more. The main problem with ML is the separation of data and compute. It takes a lot of time and energy to move data around. We need 'in memory compute'.
  - cr0sh 7y ago
    So you're in disagreement with Hinton? https://www.axios.com/artificial-intelligence-pioneer-says-we-need-to-start-over-1513305524-f619efbd-9db0-4947-a9b2-7a4c310a28fe.html https://www.axios.com/artificial-intelligence-pioneer-says-w... I've heard the argument made about "in memory compute" - and I do know that various companies have made hardware toward that direction (and some of it has very low power requirements). I just find it compelling that Hinton - at least as of 2 years ago - has this viewpoint that we should be rethinking the concept of backprop...
    - bllguo 7y ago
      that article is way too vague for agreement/disagreement to be a productive point of discussion imo. is he talking about new optimization methods?
    - jasallen 7y ago
      I'd say this is a difference of scale. Hinton's comments are meant to inspire and encourage _The Next Great Thing_. We could read the paper that broadly, but they seem to be addressing a more immediate local minima.
    - bigred100 7y ago
      Isn’t backpropagation just an efficient method for getting a gradient? It sounds like what he’s looking for an alternative to is the idea of adjusting parameters to match labeled training data.
alevskaya 7y ago
This is something we run into even when pushing transformer-style models (which can be viewed as a partial step towards some of these capsule ideas) into new areas like computer vision. There are places where you can beat convolutions in terms of parameters+FLOPs but wall-clock time is poor due to the same issues presented in this paper - there aren't good optimized self-attention kernels so you're forced to implement things in terms of existing ops - depthwise convs, etc. It's definitely a real pain when trying to bring in new primitives to the deep learning world, only compounded by the increasing diversity of accelerators in use.

by cs702 7y ago

cr0sh 7y ago

Note: I am not an expert on ML, ANNs, etc - most of what I have done has been mainly "hobby level" and MOOC learning. From what I understand, though, what GH is trying to accomplish with capsule networks (which I have tried to understand, and have yet to succeed) is optimizing (or possible removal of) backpropagation. He has noted in the past how - so far (from what I know) - there isn't a biological equivalent to backpropagation for learning. Backprop is a completely artificial mathematical construct that doesn't happen in natural systems. It is also extremely energy intensive - at least in the manner that it is currently done. So the question is - that he's hoping to answer I think - what is a proper working framework for implementing an artificial neural network that can learn, without using backpropagation (or using it differently)? I think whoever can solve this will fundamentally remake the field of ML/AI - much in the same way that backprop (and later "deep learning") did.

visarga 7y ago
It's not the backprop that is holding back ML - it takes about as much time as the forward pass and requires 3x the memory. Backprop is necessary for almost all the deep neural nets that have state of the art results, attempting to replace it would push us back a decade or more. The main problem with ML is the separation of data and compute. It takes a lot of time and energy to move data around. We need 'in memory compute'.
- cr0sh 7y ago
  So you're in disagreement with Hinton? https://www.axios.com/artificial-intelligence-pioneer-says-we-need-to-start-over-1513305524-f619efbd-9db0-4947-a9b2-7a4c310a28fe.html https://www.axios.com/artificial-intelligence-pioneer-says-w... I've heard the argument made about "in memory compute" - and I do know that various companies have made hardware toward that direction (and some of it has very low power requirements). I just find it compelling that Hinton - at least as of 2 years ago - has this viewpoint that we should be rethinking the concept of backprop...
  - bllguo 7y ago
    that article is way too vague for agreement/disagreement to be a productive point of discussion imo. is he talking about new optimization methods?
  - jasallen 7y ago
    I'd say this is a difference of scale. Hinton's comments are meant to inspire and encourage _The Next Great Thing_. We could read the paper that broadly, but they seem to be addressing a more immediate local minima.
  - bigred100 7y ago
    Isn’t backpropagation just an efficient method for getting a gradient? It sounds like what he’s looking for an alternative to is the idea of adjusting parameters to match labeled training data.

alevskaya 7y ago

This is something we run into even when pushing transformer-style models (which can be viewed as a partial step towards some of these capsule ideas) into new areas like computer vision. There are places where you can beat convolutions in terms of parameters+FLOPs but wall-clock time is poor due to the same issues presented in this paper - there aren't good optimized self-attention kernels so you're forced to implement things in terms of existing ops - depthwise convs, etc. It's definitely a real pain when trying to bring in new primitives to the deep learning world, only compounded by the increasing diversity of accelerators in use.