8 ms·
Scientists See Promise in Deep-Learning Programs
- theschwa 14y agoGeoffrey Hinton, mentioned in the article, has his class on neural networks available on Coursera https://www.coursera.org/course/neuralnets https://www.coursera.org/course/neuralnets
- ozgung 14y agoI was watching his lectures and I saw this post when I take a break. He was talking about the ups and downs in the history of Neural Nets. As far as I understand from all these lectures we're on the verge of a new up phase. Neural Nets are meaningful when they are large and deep and training such nets becomes feasible, although not immediately.
- riffraff 14y agoI started taking the class but had to take a break due to other real life occurrences. It was very enjoyable both content wise and style wise, for what my opinion count, I recommend it.
- djacobs 14y agoHinton was one of the people who invented backpropagation, which has let neural nets be as powerful as they are today. Somehow, despite his brilliance and intimate familiarity with backpropagation, his explanation of it is stunningly clear and simple. I'm thoroughly enjoying this course and recommend it to anyone who wants to build their own neural networks.
- T-A 14y ago"If you can't explain something simply, you don't know enough about it. You do not really understand something unless you can explain it to your grandmother." - Some German dude :)
- claudiusd 14y agoI'm not an expert on this, but I think this article overstates the relationship between "deep learning methods" and "neural networks". Neural nets have been around forever and, in the feed-forward case, are actually fairly basic statistical classifiers. Deep learning, on the other hand, is about using layers of classifiers to progressively recognize higher-order concepts. In computer vision, for example, the first layer of classifiers may be recognizing things like edges, blocks of color, and other simple concepts, while progressive things may be recognizing things like "arm", "desk", or "cat" from the lower-order concepts. There's a book I read a while ago that was super-interesting and digs in to how one researcher leveraged knowledge about how the human brain works to develop one of these deep learning methods: "On Intelligence" by Jeff Hawkins (http://www.amazon.com/On-Intelligence-Jeff-Hawkins/dp/B000GQLCVE/ http://www.amazon.com/On-Intelligence-Jeff-Hawkins/dp/B000GQ...)
- wookietrader 14y agoNo. All currently used deep learning algorithms are special cases of neural networks. The reason why this is called "deep" learning is that before 2006, no one knew how to efficiently train neural nets with more than 1 or 2 hidden layers. (Or could, because of computing power.) Thanks to a breakthrough by Dr Hinton, this is now the case. But all models used are neural nets. It's just that a vast amount new algorithms for training them have been developed in the last years and people came up with new ideas on how to use them. But it is all neural nets. And that's the whole beauty of it.
- mojuba 14y agoOn the other hand, there is nothing a neural net can do that a Turing machine can't do (perhaps even better?).
- wookietrader 14y agoNeural Nets cannot loop (unless they are recurrent neural nets) and are memory bound.
- jhartmann 14y agoDeep belief networks are extremely powerful, we are finally getting to the point where we don't need to do tons of feature engineering to make useful complex classifiers. Used to be you would have to spend a ton of time doing data analysis and feature extraction to get useful and robust classifiers. Of course the usefulness of those sorts of networks were limited by how well you did the feature extraction. Now you train networks with much more minimally processed data, and get great results out of them.
- mbq 14y agoSince the fall of AI, there are two groups of people in this topic -- one trying to make some reproducible, robust results with well defined algorithms and second importing random ideas from the first group onto some questionably defined ANN model and getting all the hype because of the "neural" buzzword. "Deep learning" is actually called boosting and has been around for years.
- iskander 14y agoBoosting selects successive weak learners for the same classification problem, but under a changing distribution/weighting of the input space. Deep learning stacks complex models to create increasingly abstract representations. All I can really imagine them having in common is (1) they're both families of machine learning techniques and (2) they both (roughly) involve a collection of models, albeit in very different ways.
- mbq 14y agoYou're talking about Adaboost; general boosting can use any models and the only idea there is that it adds new model to fix residuals of the current chain. BTW "increasingly abstract representation" is a perfect example of this meaningless PR ANNs are built of.
- wookietrader 14y agoNo. Deep learning is not about fixing the residuals of the current chain. Deep learning isn't even about residuals in the first place. It's about (1) finding good representations of your data (aka feature learning), (2) then adding a discriminative model on top and then (3) tuning everything. There is no relation to boosting at all.
- robrenaud 14y agoUnsupervised pre-training is fundamentally different than boosting. Boosting is a clever way of modelling a conditional distribution. The insight behind the success of pre-training is that, for many perceptual tasks, having a good model of the input (rather than the input->output mapping) is key. I have no delusion that the algorithms that work for training deep networks are anything like what the brain actually does, but I don't care. There are many tasks where deep neural nets are state of the art.
- dave_sullivan 14y agoFor those looking to learn about these techniques, I'd highly recommend the deep learning theano tutorials. Hinton has a class on Coursera--I think it would be very confusing for beginners, but it has really great material. Also, I run the "SF Neural Network Aficionados" meetup in san francisco and will be giving a workshop in January about building your own DBN in python, so feel free to check that out if you're in SF (although space was an issue last time).
- stewie2 14y agoHow is "deep learning" different from "neural network"?
- dave_sullivan 14y agoWell--two answers: 1) It's not. It's just a buzz word that people are going to use to separate the current (2006+) research from older research concerning neural networks. This is to draw a clear (and potentially self serving?) distinction between old neural networks that were discredited due to their lack of results vs current research that produces much much better results. So, it's neural networks rebranded. Oh my. 2) "traditional" neural networks and what people are using now are very different--mostly because what people are doing now actually works. Deep learning refers to deep neural networks, which take more traditional neural networks and stack them on top of each other to form a hierarchy of representations that ends up being effective for all kinds of stuff. Not that deep neural networks are a new concept--the newness is more that this is now practical rather than theoretical. So, deep learning is like 20% bullshit, 80% the real deal. Still lots of work to be done, but I think "deep learning" is a nice buzzword to describe the current state of the art as far as neural networks go. It's all neural networks--but this time it's different, haha.
- Yoshua 14y agoThe idea of having multiple levels of representation (deep learning) goes beyond neural networks. A good example is the recent work (award-winning at NIPS 2012) on sum-product networks, which are graphical models whose partition function is tractable by construction. Several important things have been added since 2006 (when deep learning was deemed to begin) to the previous wave of neural networks research, in particular powerful unsupervised learning algorithms (which allow very successful semi-supervised and transfer learning - 2 competitions won in 2011), often incorporating advanced probabilistic models with latent variables, a better understanding (although much more remains to be done) of the optimization difficulty of training gradient-based systems through many composed non-linearities, and other improvements to regularize better (such as the recent dropouts) and to rationally and efficiently select hyper-parameters (random sampling and Bayesian optimization). It is also true that sheer improvements in computing power and amounts of training data are in part responsible for the impressively good results recently obtained in speech recognition (see recent New York Times article, 24 nov, J. Markoff) and object recognition (see NIPS 2012 paper by Krizhesky et al).
- taliesinb 14y agoI played around for a while with writing an RBM learner in Go (RBMs are a particular instance of deep learning which Hinton specializes in). More an experiment than anything else, but for anyone who is interested: https://github.com/taliesinb/gorbm https://github.com/taliesinb/gorbm. I don't claim there aren't bugs, and there is no documentation. The consensus I've picked up from AI-specializing friends is that there are a lot of subtle gotchas and tricks (which Hinton and friends know about but don't necessarily advertise) without which RBMs are a non-starter for many problems. Which I suppose is pretty much standard for esoteric machine learning.
- radarsat1 14y agoAre there any good C++ or Python SciPy libraries for building and training deep learning networks?
- osdf 14y agoMentioned somewhere else: http://deeplearning.net/software/theano/ http://deeplearning.net/software/theano/ with a tutorial http://deeplearning.net/tutorial http://deeplearning.net/tutorial Not C++ or Python, but lua with lots of stuff: torch 7 [[http://www.torch.ch/] http://www.torch.ch/]]
- jhartmann 14y agoThere is a C++/Cuda library with a python frontend that I am starting to play with that is from one of the guys who works with Hinton. It is written by Alex Krizhevsky and has lots of tools for training feed forward networks with lots of different connection topologies and neuron types. If I am not mistaken this was the library that was used in the recent Kaggle drug competition that is referenced in the article. There is some good starting point documentation here as well to look into, as long as you know enough about the mechanics of Artificial Neural Networks it has some really interesting stuff in there. Here is the link: http://code.google.com/p/cuda-convnet/ http://code.google.com/p/cuda-convnet/
- mturmon 14y agoPegged to the NIPS conference next week: http://nips.cc/Conferences/2012/ http://nips.cc/Conferences/2012/
- bravura 14y agoSince I see some misunderstanding about deep learning, let me explain the fundamental idea: It's about reusing intermediate work. The intuition is let's say I told you to write a complicated computer program. Let's say I told you that you could use routines and subroutines, but you couldn't use subsubroutines, or deeper levels of abstraction. In this restricted case, you could write any computer program, but you would have to use a lot of code-copying. With arbitrary levels of abstraction, you could do code reuse much more elegantly, and your code would be more compact. Here is a more formal description: If you have a complicated non-linear function, you can describe it similarly to a circuit. If you restrict the depth of the circuit, you can in principle represent any function, but you need a really wide (exponentially wide) circuit. This can lead to overfitting. (Occam's Razor) By comparison, with a deep circuit, you can represent arbitrary functions compactly. Standard SVMs and random forests can be shown, mathematically, to have a limited number of layers (circuit depth). It turns out that expressing deep models using neural networks is quite convenient. I gave an introduction to deep learning in 2009 that describes these intuitions: http://vimeo.com/7977427 http://vimeo.com/7977427
- tgflynn 14y agoIf you restrict the depth of the circuit, you can in principle represent any function, but you need a really wide (exponentially wide) circuit. Are you sure it's exponential ? If you look at binary functions (ie. boolean circuits) any such function can be represented by a single layer function whose size is linear in the number of gates of the original function (I think it's 3 or 4 variables per gate) by converting to conjunctive normal form. Of course it's not obvious that a similar scaling exists for non-binary functions but I'd be a bit surprised if increasing depth led to an exponential gain in representational efficiency.
- bravura 14y agoI am not sure in the sense of: If I were dropped on a desert island, I could derive a water-tight proof of this result from scratch. I am confident, though, based upon my reading of secondary sources written by people that I trust. From one of Bengio's works (http://www.iro.umontreal.ca/~bengioy/papers/ftml.pdf http://www.iro.umontreal.ca/~bengioy/papers/ftml.pdf): "More interestingly, there are functions computable with a polynomial-size logic gates circuit of depth k that require exponential size when restricted to depth k − 1 (Hastad, 1986)."
- gdahl 14y agoI was involved in the speech recognition work mentioned in the article and I led the team that won the Merck contest if anyone has any questions about those things. I also spend some time answering any machine learning question I feel qualified to answer at metaoptimize.com/qa
- Aron 14y agoI worked on the Netflix prize and haven't learned anything since then. There the RBM (or modified version per ruslan's paper) performed very well but not substantially better than the linear models (in apples to apples comparison.. ignore the time-dimension and peeking at the contents of the quiz\test set). And as I recall no one really made any progress with deeper networks on that problem. Has anything been learned since then that would suggest progress there? I also don't recall anyone successfully incorporating the date of the rating into the RBM. Mostly this was useful in other models because on any particular day people would just bias their ratings up or down a bit. But also, as one can imagine, over the course of a year or two their tastes would change. Is it straightforward to include that time dimension into RBMs, and if so, is that a recently discovered technique?
- arekp 14y agoThe Netflix Prize winners had a few RBM models that used the dates. Regarding the DBM - I also tried to use more than one layer, and without success. I tried out 3-layer and 4-layer autoencoders (can be called 1.5-layer and 2-layer DBM), with initialization by stacked RBMs or without it. It did not work well probably because: a) the model was inaccurate, and b) the learning method proposed for DBM was not completely correct. Intuitively, the right DBM-like model with the right learning method should have a chance to improve something on the Netflix task. I found some improvement though (rather learning time than accuracy) in the standard RBMs. Instead of using CD, I split the weights into two sets, creating a directed RBM version. The "up" weights from the visible nodes to hidden are learned with CD with T=1. The "down" weights are learned to best fit the visible nodes, using the hidden nodes as predictors. The hidden nodes generated by CD T=1 are good enough, and we do not need additional iterations with increased T.
- 14y ago
- teeja 14y agoIs there a good place to plug-in to get an overview of what (has been and is) going on in this area, without having to dive in all the way? An overview of the concepts, not the nuts and bolts, not the heavy-lifting.
- pilooch 14y agothe one overview I've found the most useful is http://www.youtube.com/watch?v=ZmNOAtZIgIk http://www.youtube.com/watch?v=ZmNOAtZIgIk (Bay Area Vision Meeting: Unsupervised Feature Learning and Deep Learning by Andrew Ng in April 2011).
- pootch 14y agoNone of this is a breakthrough. This article was totally doing what every AI article has done since the term has existed. Claim that a statistical variance in a very specific field of research like oh say facial recognition or whatever means that AI itself has moved forward. It has not. Deep learning is not going anywhere. Its a false premise for algorythms that have trained neural nets better over a significant amount of time in fields that people care about. It is not a solution, it is not new technology. Molecular structures are pure math and computers do that pretty easily last time I checked.
- Create 14y agoThe students were also working with a relatively small set of data; ANN-s are overfitted more often than not.
- mikecane 14y agoCan someone contrast what's in that article with what Jeff Hawkins' Numenta is attempting?