6 ms·

Stacked Approximated Regression Machine: A Simple Deep Learning Approach

123 points by kartikkumar 10y ago

cs702 10y ago
As far as I understand this, these guys claim they can train convolutional and many other types of deep neural nets faster by pretraining each layer with a new unsupervised technique via which the layer sort of learns to compress its inputs (a local optimization problem), and then they fine tune the whole network end-to-end with supervised SGD and backpropagation as usual. They have not released code, so no one else has replicated this yet -- as far as I know. If the claim holds, the implication is that layers can quickly learn much of what they need to learn locally, that is, without requiring backpropagation of gradients from potentially very distant layers. I can't help but wonder if this opens the door for more efficient asynchronous/parallel/distributed training of layers, potentially leading to models that update themselves continuously (i.e., "online" instead of in a batch process). I wouldn't be surprised if the claim holds. There is mounting evidence that standard end-to-end backpropagation is a rather inefficient learning mechanism. For example, we now know that deep neural nets can be trained with approximate gradients obtained by shifting bits to get the sign and order of magnitude of the gradient roughly right.[1] In some cases it's even possible to restrict learning to use binary weights.[2] More recently, we have learned that it's possible to use "helper" linear models during training to predict what the gradients will be for each layer, in-between true-gradient updates, allowing layers to update their parameters locally during backpropagation.[3] Finally, don't forget that in the late 2000's, AI researchers were doing a lot of interesting work with unsupervised layer-wise training (e.g., DBNs composed of RBMs, stacked autoencoders).[4] This is a fascinating area of research with potentially huge payoffs. For example, it would be really neat if we find there's a "general" algorithm via which layers can learn locally from inputs continuously ("online"), allowing us to combine layers into deep neural nets for specific tasks as needed. [1] https://arxiv.org/abs/1510.03009 https://arxiv.org/abs/1510.03009 [2] https://arxiv.org/abs/1602.02830 https://arxiv.org/abs/1602.02830 [3] https://deepmind.com/blog#decoupled-neural-interfaces-using-synthetic-gradients https://deepmind.com/blog#decoupled-neural-interfaces-using-... [4] https://www.iro.umontreal.ca/~lisa/pointeurs/TR1312.pdf https://www.iro.umontreal.ca/~lisa/pointeurs/TR1312.pdf EDITS: Expanded the original comment so it conveys better what I actually meant to write, while keeping language as casual and informal as possible. Also, I softened the tone of my more speculative observations.
- plusepsilon 10y ago
  Thanks for the background info. If the layers are learned locally, can it take into account interactions between layers?
  - orangutango 10y ago
    Locally != independently; the training process is recursive, so interactions between layers are present. From the paper: "The parameters of the entire SARM are solved recursively. The current ARM’s parameters are calculated using the output from the previous ARM. Then, the output of the current ARM is fed into the subsequent ARM (or the classifier, if the current one is the last ARM), as its input."
- joe_the_user 10y ago
  The paper itself is fairly sparse but references a number of approaches for more-quickly-learning neural-net-related learning systems from ~2013 (PCAnet, SCATnet,etc). The paper presents these and other approaches as being instances of a classical, general form, regularized regression but with the "stacked" property involving each layer iterating however many times and then the next layer changing parameters (or features) and further iterating. From my barely-informed viewpoint, this sounds like a fascinating way to unify the earlier efforts and one which could yield a variety of other approaches - even if the particular variation they use doesn't work out. But I assume lots of more-informed people are going to be looking at this.
  - joe_the_user 10y ago
    Note: even though the paper talks about Approximate Regression Machine layers and uses equations that look sort-of like the equations of regularized regression, the layers aren't about regularized regression but about sparse dictionary encoding, a quite different approach.
- jcannell 10y ago
  This is a pretty good summary, but to note this is not a new unsupervised technique - it's a sparse coding method, and sparse coding has been around for more than a decade. Summarizing this paper without mentioning the word sparse coding seems wrong somehow. ;)
imurray 10y ago
I'm going to check back later to see if anyone manages to reproduce it. Perhaps by the time it's presented at NIPS. A twitter conversation reflecting some scepticism, but agreeing it would be interesting if it all checks out: https://twitter.com/fchollet/status/771862837819867136 https://twitter.com/fchollet/status/771862837819867136
- cs702 10y ago
  I'm less skeptical than fchollet (creator of Keras, for those here who don't know), but agree that we need to wait until the usual suspects at Google, FaceBook, Toronto, Montreal, Stanford, etc. have replicated this. In all likelihood the team will release code soon, either before or after NIPS, so we will all be able to check things out for ourselves.
  - cs702 10y ago
    One of the authors, Zhangyang Wang, just wrote this on his personal page: "We have discussed and decided to work on a software package release, perhaps accompanying it with a more detailed technical report in the future. Once the software package is ready, we will update everybody." http://www.atlaswang.com/ http://www.atlaswang.com/
    - joe_the_user 10y ago
      Paper withdrawn https://arxiv.org/abs/1608.04062 https://arxiv.org/abs/1608.04062 It's kind of an odd thing. I (random non-academic amateur) actually spent a bunch of time trying to parse the paper, which was kind of a combination of interesting ideas and incomprehensible ambiguities. One real academic researcher also put some time into it. The good part of the paper is explained here. My guess is the problem is going from ARM to SARM. http://gabgoh.github.io/SARG/ http://gabgoh.github.io/SARG/ While I'm sure most people involved think of the experience as a wash, I feel like I learned a bunch about deep learning in the process. PS, also sad that the author did this.
      - gabrielgoh 10y ago
billconan 10y ago
is there a book you can recommend about the fundamentals (like sparse coding) to understand papers like this?
- jlg23 10y ago
  http://www.scholarpedia.org/article/Sparse_coding http://www.scholarpedia.org/article/Sparse_coding
  - nojvek 10y ago
    This was a great read. Thanks.
- kleiba 10y ago
  http://www.deeplearningbook.org http://www.deeplearningbook.org
- gabrielgoh 10y ago
  The fundamentals are pretty easy. I wrote a blog post on this, let me know if it helps http://gabgoh.github.io/SARG/ http://gabgoh.github.io/SARG/
  - billconan 10y ago
    Thank you, I will read it! (hey! I'm also from UCDavis!)
    - gabrielgoh 10y ago
      woop, a fellow aggie! feel free to email me if you have any questions, i want to make the post as clear as possible
nl 10y ago
Update on this: it has been withdrawn: https://arxiv.org/abs/1608.04062 https://arxiv.org/abs/1608.04062

123 points by kartikkumar 10y ago

cs702 10y ago

As far as I understand this, these guys claim they can train convolutional and many other types of deep neural nets faster by pretraining each layer with a new unsupervised technique via which the layer sort of learns to compress its inputs (a local optimization problem), and then they fine tune the whole network end-to-end with supervised SGD and backpropagation as usual. They have not released code, so no one else has replicated this yet -- as far as I know. If the claim holds, the implication is that layers can quickly learn much of what they need to learn locally, that is, without requiring backpropagation of gradients from potentially very distant layers. I can't help but wonder if this opens the door for more efficient asynchronous/parallel/distributed training of layers, potentially leading to models that update themselves continuously (i.e., "online" instead of in a batch process). I wouldn't be surprised if the claim holds. There is mounting evidence that standard end-to-end backpropagation is a rather inefficient learning mechanism. For example, we now know that deep neural nets can be trained with approximate gradients obtained by shifting bits to get the sign and order of magnitude of the gradient roughly right.[1] In some cases it's even possible to restrict learning to use binary weights.[2] More recently, we have learned that it's possible to use "helper" linear models during training to predict what the gradients will be for each layer, in-between true-gradient updates, allowing layers to update their parameters locally during backpropagation.[3] Finally, don't forget that in the late 2000's, AI researchers were doing a lot of interesting work with unsupervised layer-wise training (e.g., DBNs composed of RBMs, stacked autoencoders).[4] This is a fascinating area of research with potentially huge payoffs. For example, it would be really neat if we find there's a "general" algorithm via which layers can learn locally from inputs continuously ("online"), allowing us to combine layers into deep neural nets for specific tasks as needed. [1] https://arxiv.org/abs/1510.03009 https://arxiv.org/abs/1510.03009 [2] https://arxiv.org/abs/1602.02830 https://arxiv.org/abs/1602.02830 [3] https://deepmind.com/blog#decoupled-neural-interfaces-using-synthetic-gradients https://deepmind.com/blog#decoupled-neural-interfaces-using-... [4] https://www.iro.umontreal.ca/~lisa/pointeurs/TR1312.pdf https://www.iro.umontreal.ca/~lisa/pointeurs/TR1312.pdf EDITS: Expanded the original comment so it conveys better what I actually meant to write, while keeping language as casual and informal as possible. Also, I softened the tone of my more speculative observations.

plusepsilon 10y ago
Thanks for the background info. If the layers are learned locally, can it take into account interactions between layers?
- orangutango 10y ago
  Locally != independently; the training process is recursive, so interactions between layers are present. From the paper: "The parameters of the entire SARM are solved recursively. The current ARM’s parameters are calculated using the output from the previous ARM. Then, the output of the current ARM is fed into the subsequent ARM (or the classifier, if the current one is the last ARM), as its input."
joe_the_user 10y ago
The paper itself is fairly sparse but references a number of approaches for more-quickly-learning neural-net-related learning systems from ~2013 (PCAnet, SCATnet,etc). The paper presents these and other approaches as being instances of a classical, general form, regularized regression but with the "stacked" property involving each layer iterating however many times and then the next layer changing parameters (or features) and further iterating. From my barely-informed viewpoint, this sounds like a fascinating way to unify the earlier efforts and one which could yield a variety of other approaches - even if the particular variation they use doesn't work out. But I assume lots of more-informed people are going to be looking at this.
- joe_the_user 10y ago
  Note: even though the paper talks about Approximate Regression Machine layers and uses equations that look sort-of like the equations of regularized regression, the layers aren't about regularized regression but about sparse dictionary encoding, a quite different approach.
jcannell 10y ago
This is a pretty good summary, but to note this is not a new unsupervised technique - it's a sparse coding method, and sparse coding has been around for more than a decade. Summarizing this paper without mentioning the word sparse coding seems wrong somehow. ;)

imurray 10y ago

I'm going to check back later to see if anyone manages to reproduce it. Perhaps by the time it's presented at NIPS. A twitter conversation reflecting some scepticism, but agreeing it would be interesting if it all checks out: https://twitter.com/fchollet/status/771862837819867136 https://twitter.com/fchollet/status/771862837819867136

cs702 10y ago
I'm less skeptical than fchollet (creator of Keras, for those here who don't know), but agree that we need to wait until the usual suspects at Google, FaceBook, Toronto, Montreal, Stanford, etc. have replicated this. In all likelihood the team will release code soon, either before or after NIPS, so we will all be able to check things out for ourselves.
- cs702 10y ago
  One of the authors, Zhangyang Wang, just wrote this on his personal page: "We have discussed and decided to work on a software package release, perhaps accompanying it with a more detailed technical report in the future. Once the software package is ready, we will update everybody." http://www.atlaswang.com/ http://www.atlaswang.com/
  - joe_the_user 10y ago
    Paper withdrawn https://arxiv.org/abs/1608.04062 https://arxiv.org/abs/1608.04062 It's kind of an odd thing. I (random non-academic amateur) actually spent a bunch of time trying to parse the paper, which was kind of a combination of interesting ideas and incomprehensible ambiguities. One real academic researcher also put some time into it. The good part of the paper is explained here. My guess is the problem is going from ARM to SARM. http://gabgoh.github.io/SARG/ http://gabgoh.github.io/SARG/ While I'm sure most people involved think of the experience as a wash, I feel like I learned a bunch about deep learning in the process. PS, also sad that the author did this.
    - gabrielgoh 10y ago

billconan 10y ago

is there a book you can recommend about the fundamentals (like sparse coding) to understand papers like this?

jlg23 10y ago
http://www.scholarpedia.org/article/Sparse_coding http://www.scholarpedia.org/article/Sparse_coding
- nojvek 10y ago
  This was a great read. Thanks.
kleiba 10y ago
http://www.deeplearningbook.org http://www.deeplearningbook.org
gabrielgoh 10y ago
The fundamentals are pretty easy. I wrote a blog post on this, let me know if it helps http://gabgoh.github.io/SARG/ http://gabgoh.github.io/SARG/
- billconan 10y ago
  Thank you, I will read it! (hey! I'm also from UCDavis!)
  - gabrielgoh 10y ago
    woop, a fellow aggie! feel free to email me if you have any questions, i want to make the post as clear as possible

nl 10y ago

Update on this: it has been withdrawn: https://arxiv.org/abs/1608.04062 https://arxiv.org/abs/1608.04062