14 ms·
What is the difference between deep learning and usual machine learning?
- taneq 10y agoDeep learning is machine learning involving neural nets with more than one hidden layer.
- tomp 10y ago... and a method to train the hidden layers efficiently.
- taneq 10y agoWell, therein lies the rub, yes.
- _delirium 10y agoThat's arguable; most successful applications of deep learning aren't particularly efficient on the training side, which is why they come out of research groups that have Google-sized compute clusters available, typically with big arrays of recent GPUs. I suppose they're efficient in the sense of not requiring more computation than exists on earth. There have been a bunch of algorithmic advances as well, but some advances have simply been those increases in compute power: there are techniques that didn't really work in 1985, where the technique or a close variant now works, mainly because we didn't the big arrays of GPUs in 1985 that we do now. As these get scaled up, the algorithmic advances are being found to not even be necessary in some cases. For a period it was thought that autoencoder pretraining was the big algorithmic breakthrough that made it feasible to train many-layered neural networks, but a number of recent applications no longer bother with the autoencoder pretraining.
- Kip9000 10y agoTo do increasingly fancy things with feed forward neural networks, we require increasingly many layers. But with many layers, it becomes increasingly difficult to train it (vanishing gradient, explosive growth of connections, etc). But Hinton et al. found a few tricks to go past those obstacles and that's what 'deep learning' encompasses. Current state of the art of deep learning is at 150 odd layers with a new trick called 'residual learning' by Microsoft Research China.
- taneq 10y agoI didn't say it was easy to do, if anything I was just taking a pot shot at the laziness of using the term as a label for all of modern machine learning. I've been studying up on deep neural nets recently though and there are a bunch of very cool tricks (and I don't mean that pejoratively, I think real intelligence consists of just such a grab-bag of tricks), which make it effective in a wide range of cases.
- nabla9 10y agoMost definitions of deep learning don't include world "neural network". They are synonymous for multi layer neural network techniques for now, but not necessarily in the future. For example: >Deep-learning methods are representation-learning methods with multiple levels of representation, obtained by composing simple but non-linear modules that each transform the representation at one level (starting with the raw input) into a representation at a higher, slightly more abstract level. Deep Learning, Nature Volume 521 issue 7553, 2015, LeCun, Yann; Bengio, Yoshua; Hinton, Geoffrey
- wwwdonohue 10y agoI'm going out on a limb here but I'm guessing if this explanation makes sense to you, you don't need to be told the difference between deep and usual machine learning. Could be wrong!
- brians 10y agoNope. I have a 1990s understanding of perceptrons; this helped show me a bit of what the current rage is about.
- dreamcompiler 10y agoNot true. I know quite a bit about three-layer networks and backprop but had been puzzled about how people were training networks with more layers. This article was helpful.
- logicallee 10y agoIt sounds like you're implying, but don't want to state, that the explanation is not clear enough for outsiders (such as yourself?) If I have read your comment correctly, I'll say it for you: as an outsider, I read this carefully until I gave up because it was too technical, which happened right at the very top, in the third paragraph: >Those hidden layers normally have some sort of sigmoid activation function (log-sigmoid or the hyperbolic tangent etc.). For example, think of a log-sigmoid unit in our network as a logistic regression unit that returns continuous values outputs in the range 0-1 All this implies I know all about multi-layer perceptrons - and I don't. I can't follow the instructions to "think of a log-sigmoid unit in our network as a logistic regression unit" because I don't know what those terms mean. Just as I would give up on a recipe if I got to an instruction I didn't know. For example, if I read: >Glaze the meringue with a blow torch, or briefly in a hot oven. Yeah, uh, no... I don't even know what glazing means, or what is "briefly in a hot oven". So I just stop reading. When I'm instructed to do something I can't, I go look at something else unless I'm feeling very adventurous.[1] This blog post isn't written at my level. -- [1] as a last hoorah I'll open a tab and Google https://www.google.com/search?q=what+is+glazing https://www.google.com/search?q=what+is+glazing - likewise I tried https://www.google.com/search?q=what+is+a+multilayer+perceptron https://www.google.com/search?q=what+is+a+multilayer+percept... but decided after reading the Wikipedia link that it was too "deep" for me.
- easytiger 10y agoBoth are facile bullshit designed to eat up CPU cycles. Which, incidentally is what major corporations are basing their business models on selling. I've had clients come to me and say they want to use machine learning. When I ask what for, it becomes very awkward.
- MichailP 10y agoCan someone say a bit about minimum number of layers and neurons needed for structure to be called artificial neural network? I have seen papers with something like 10 neurons (which is very very tiny but authors claim it does the job), while on the other hand there are Google sized ANNs. Thanks :)
- florianletsch 10y agoYou can call it an artificial neural network (ANN) as soon as you have two or more artifical neurons connected to each other. As simple as that. A small amount of neurons might already solve some problem you're having. The XOR problem can be learned by 4 neurons connected to each other. When you want raw images or similar as an input and have it be classified into 100 classes (e.g. look up CIFAR-10 or CIFAR-100), you will need an architecture with many more neurons. After all, ANN are simply a tool. Depending on the task, that tool might need to more elaborate. And when you have all those different possible architectures, you want a common way of naming them. Labels such as Deep Learning are simply nomenclature of talking about certain groups of artificial neural networks.
- PeterisP 10y agoANN is a description about structure, not size. XOR (an input layer of 2 neurons, 1 hidden layer of usually 2 neurons, and an output layer of 1 neuron) is a classic toy example of an ANN. The minimal would probably be something like a NOT gate - input layer with 1 neuron connected directly to the output layer with 1 neuron. In practice you either use whatever size your problem needs (for small problems) or match the network size to match the RAM amount in the video cards you're using.
- rasbt 10y agoTechnically, you can say that even a single perceptron unit is a artificial neural network -- a single-layer ANN. Multi-layer neural networks (such as a multi-layer perceptron) have at least one input layer, one hidden layer, and one output layer.
- MichailP 10y agoThanks for all replies. For some reason I felt that ANN has to be large to perform something useful, and small ~10 neurons ANNs seemed like a toy compared to ~1e6 nets. But, as you pointed out, if it does the job then it is good enough, no matter the count of neurons.
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- phinance99 10y agoFeature Learning. Deep Neural Networks can learn features from essentially raw data. Usual machine learning starts with features engineered manually. DNNs also learn to predict from the features they learn, so you cold say (very roughly) "DNN = usual machine learning + feature learning". In practice manually engineering features is a time-consuming "guess-and-check" process which benefits from domain expertise. Feature Learning, otoh, is more automatic and benefits from data, computing resources, and optimization algorithms.
- facepalm 10y agoThere are lots of other unsupervised learning methods in machine learning.
- eshvk 10y ago> Deep Neural Networks can learn features from essentially raw data. Usual machine learning starts with features engineered manually. What does this mean? What is "raw data" and what is a "feature engineered manually"?
- eegilbert 10y agoOne example is this recent paper on learning (somewhat) high-level attributes of text from character streams alone (i.e., without telling the convolutional networks that things like words and punctuation exist). https://arxiv.org/abs/1502.01710 https://arxiv.org/abs/1502.01710
- YeGoblynQueenne 10y ago>> We show that temporal ConvNets can achieve astonishing performance Yay, astonishing performance! I'm totally gonna waste half an hour of my life to read about what awesome badassery convnets are! Because that sounds so objective! /snark
- hacker42 10y agoTens of thousands of engineers (audio, vision, linguists etc.) spent millions of hours for billions of dollars in the past 30 years to invent algorithms that reliably tell us something about a bunch of data. For example, an corner feature algorithm (such as SIFT) can extract the locations of corners in an image and characterize them. This is essential to many kinds of information processing tasks because we want to apply the same algorithm to different data (generalization), so we kind of need an interface to the data. This interface is called a feature (or feature algorithm, feature extractor or feature-descriptor). All of this work (some of these papers have on the order of ten thousands of citations) is now obsolete because you can start with a random initialization of the weights of a neural network and iteratively improve the weights using backprop for any kind of task. All you need a measure of improvement that is relatively smooth and differentiable with respect to the network weights. What is surprising is that the circuits and programs within reach of backprop training of fully connected neural networks are actually astonishingly good at what they do. But ultimately, this is maybe not so surprising given that our brains do something similar all the time.
- S4M 10y agoSlightly OT, but what exactly is the difference between machine learning and statistics?
- vcdimension 10y agofunding
- capnrefsmmat 10y agoYou may enjoy Leo Breiman's famous article "Statistical Modeling: The Two Cultures": http://projecteuclid.org/euclid.ss/1009213726 http://projecteuclid.org/euclid.ss/1009213726 What he calls "algorithmic modeling" is what I see as machine learning style thinking. Naturally there's a lot of overlap and people in stats and ML often do related work (our statistics department, at CMU, collaborates a lot with the machine learning department), but there's a basic mindset difference.
- S4M 10y agoReally interesting article. After reading it, it seems that statistics are about devising a model (like y=ax+b+epsilon) and then fitting it on the data while machine learning algorithms are about letting the data make their own model).
- nfoz 10y agoI think a statistical view of that dichotomy is "parametric statistics" vs. "non-parametric statistics": https://en.wikipedia.org/wiki/Nonparametric_statistics https://en.wikipedia.org/wiki/Nonparametric_statistics I see "machine learning" as exploring computational approaches to non-parametric statistics. The idea of data-driven model-estimation is not outside the scope of "statistics".
- kgwgk 10y agoMachine learning Statistics network, graphs model weights parameters learning fitting generalization test set performance supervised learning regression/classification unsupervised learning density estimation, clustering large grant = $1,000,000 large grant = $50,000 nice place to have a meeting: nice place to have a meeting: Snowbird, Utah, French Alps Las Vegas in August http://brenocon.com/blog/2008/12/statistics-vs-machine-learning-fight/ http://brenocon.com/blog/2008/12/statistics-vs-machine-learn...
- ascendingPig 10y agoI don't buy it. Even DL researchers will point to representation learning systems like word2vec, a shallow NN, as as examples of the success of DL approaches. My take: "Deep Learning" is performative (https://en.wikipedia.org/wiki/Performative_utterance https://en.wikipedia.org/wiki/Performative_utterance). An approach falls under the header of "Deep Learning" when used or developed by someone who identifies as a Deep Learning Researcher.
- Hondor 10y agoIt has an unusual use of "we" instead of "us" in a few places. "... that helps we to detect features ...". I wonder if the author originally used "you" and "your" then did a search and replace to "we" and "our" because 2nd person sounds too informal? I notice the former two words don't appear anywhere. Re-reading it with "you" and "your" and I realize that it works just as well. I guess it's really just a fashion which person you use. It used to be 3rd for academic writing. Now it's 1st, but I guess the most informal way is still 2nd.
- nilved 10y agoCould they be ESL?
- deleted 10y ago[deleted]
- pmoriarty 10y agoThis may come as a shock to the layperson, but there's more to artificial intelligence than neural nets, and many non-NN AI approaches could arguably be said to "learn", and are thus "machine learning" too, depending on your definition. I'm thinking of evolutionary algorithms, various other biologically inspired computation techniques (of which NN's are but one example), more traditional AI techniques such as expert systems, and a whole host of stochastic, non-biologically inspired algorithms. I'm not super familiar with NN's myself, so I can't say whether the gigantically disproportionate attention from the media and the research community is deserved based on actual superiority in effectiveness of NN's compared to other techniques, or whether they're used and talked about mostly because that's what most people know. It would be interesting to hear the thoughts of a non-NN AI researcher regarding this.
- NhanH 10y agoI've been looking to learn more about non-machine learning AI research for the past couple of months (essentially, something that is the spiritual successor to the GOFAI/expert system path instead of the currently popular statistical machine learning path). For some reasons it's quite hard. I'd appreciate if anyone can point me to the state of the research in those areas? (Conferences, research groups' names etc).
- sbw1 10y agoClassic non-ML AI research these days sometimes tends to be baked into the background of systems where deep learning is the star-- e.g., all of the search techniques used in AlphaGo are interesting in their own right, but aren't really where the innovation is there. There are researchers actively looking at more holistic AI systems that aren't just the core learning part (often called cognitive architectures). For instance, take a look at Soar: http://soar.eecs.umich.edu/ http://soar.eecs.umich.edu/ It under active development, and has been since the 80s. Soar is very much a spiritual successor to GOFAI (or maybe still is GOFAI), but does incorporate statistical machine learning-- much of the recent work has been around integrating reinforcement learning with symbolic decision-making. ACT-R is another cognitive architecture under active development, but is more used for cognitive modeling (i.e. psychology research): http://act-r.psy.cmu.edu/ http://act-r.psy.cmu.edu/ You can find other related systems by searching for "cognitive architecture", but it is unfortunately a field that attracts a lot of proposals/'designs' for things that never get properly implemented (the above being two notable exceptions).
- vikingyc 10y agoThere is no difference.
- B1FF_PSUVM 10y agoNo, the hole they dug now is deeper. Better spades. It will also take 30 years to get out of, as usual. Winter is coming.
- mcv 10y agoThere is no such thing as "usual machine learning". There are many machine learning techniques, Neural Nets is one of them, and deep learning is a very powerful way of training deep neural networks. Deep Learning is absolutely interesting and powerful, but it's not like all the other techniques are all the same, or necessarily less powerful, for that matter. Different techniques have different applications.
- shpx 10y agoThe words "deep learning" don't refer to a way of training neural nets. It's a marketing term for neural nets that have "a lot" of layers. Neural nets have been around for over 30 years, calling them deep learning was a smart re-branding move.
- mcv 10y agoFrom what little I've read, deep learning is a new way of training nets. Deep neural nets have also been around for 30 years, but backpropagation, their traditional learning algorithm, isn't as efficient in training the middle layers.
- p1esk 10y agoYou should keep reading. Backprop is still the best, and pretty much the only way to train neural nets (deep or not). Other ways exist (e.g. weight perturbation), but no one uses them. EDIT: I have to clarify: backprop is just one half of the training algorithm, you also gradient descent, and there are many variants of it.
- mcv 10y agoMan, I've been out of Neural Nets a long time. I studied them back in the early 1990s. Perceptrons and backpropagation. Didn't really keep up with the state of the art, I'm afraid. Maybe I should catch up.
- hellofunk 10y agoNone of these techniques are good in the context of human-style learning, where a child (or an adult) can learning how to do something new from only a few cases of trial and error, whereas a machine using any modern technique requires an immense number of examples to see first (often millions) before it can do the equivalent task but with less accuracy. NN and others are totally awesome, but we are so far away from true learning that is normally meant in the human context.
- selectron 10y agoPeople can't learn things which are truly new from only a few cases of trial and error either. For example, it takes a long time to learn a foreign language or become an expert at chess. Humans mainly learn things better because our training data is usually larger than that of the model for the task at hand.
- hellofunk 10y agoSuppose you want a computer vision model to start recognizing a particular object (and its subtle variations) in photos, but it has never seen this object before. How many of these objects must the model be trained on before it can generalize and recognize the object again? Now show a child some bizarre object just once, and see if it can find the object in any photos, even if the object is at different angles or subtly varied. The child will certainly do better, off just one training example.
- sukilot 10y agoHumans have a semi-hardwired architecture that evolved for classic human needs but is less general, less powerful , and less useful than artificial neural nets.
- hellofunk 10y ago>less general, less powerful , and less useful than artificial neural nets yet still much more capable in most cases. Handwriting recognition, for example. How many examples must a NN see to start accurately arriving at valid outputs for new handwritten cases? Now, how many examples must a human see to learn a particular letter before she starts recognizing it with 100% accuracy?
- platz 10y ago"We then connect those "receptive fields" (for example of the size of 5x5 pixel) with 1 unit in the next layer" I felt this could have been expanded on.. still not sure how the sliding windows map into units.
- rasbt 10y agoI agree with you, it could surely expand this answer (I am the person who wrote this little answer). Since this was originally just an answer to a question I answered via email (if I remember correctly), I didn't go into too much depth regarding ConvNets, because I just wanted to answer this question "briefly" in this mail. But if there's a demand for that (if it's useful to others) I may end up writing a tutorial on ConvNets one day :). However, there's already an excellent one out there, I highly recommend Dumoulin & Visin's "A guide to convolution arithmetic for deep learning" at https://arxiv.org/abs/1603.07285 https://arxiv.org/abs/1603.07285
- vonklaus 10y agoi think one of the one kf the problems is that articles like this; not just about ML but advanced CS and science concepts, answer a question at a complexity well above any person who would really ask it. By that I mean, if you could follow the casual explanation and complex concepts that are assumed that the reader grasps, you probably already know the difference between deep learning & ML and thus you read it out of interest to compare your view or get more information to add to your core base on the topics. That is my 0.02, I couldn't follow this article without doing ancilliary research/wikipedia lookups at least.
- rasbt 10y agoHi @vonklaus, thanks for the feedback. This was actually not intended to be an "article" but more like an answer to a targeted question (I am the author of this little write-up). Basically, someone asked me this specific question some time ago (I think via email), and I answered it with this person's background in mind. Then, I generalized it a bit more and added it to the FAQ section in the GitHub repo in hope that it is also helpful to others. It's really more like a quick overview, idea, explanation in contrast to a fully fleshed-out blog article :)
- vonklaus 10y agohey thanks for the response, I am sure it is quite well done to your target audience. Do you know of any higher level material that provides a good outline conceptually but only assumes a really general knowledge? This is a bit above my comfort zone :). I do really like, i think Joel Gruus, writing. His fizzbuzz with tensor flow was hilarious. I am unfortunately not particularly gifted in mathematics so libraries like that will likely be the furthest ill go into ML/DL, oh btw whats the difference ;). jk cheers
- rasbt 10y agoyou mean material specific to deep learning that is more general and less math heavy? Hm, that's a good question, the resources I'd reference are all a bit math heavy. However, don't be afraid of diving into TensorFlow, it's really a nice library that takes care of all the tedious, mathematical details. E.g., in contrast/addition to NumPy (leaving out the comp. efficiency part out of the discussion for now), it already implements several optimzation algorithms, so you wouldn't have to worry about implementing backpropagation from scratch or so. Sure, it still requires a bit of linear algebra, but it's really more straight-forward than it seems at first glance :). Maybe, you'd be interested in Keras (http://keras.io); http://keras.io); it's a wrapper around Theano and TensorFlow which provides a really intuitive interface for building neural nets! Haha, btw. I really enjoyed Joel Gruus, post ;)
- deleted 10y ago[deleted]
- boltzmannbrain 10y agoTrying to explain deep learning in a single FAQ response is silly. I submitted a PR with an improved question and answer: https://github.com/rasbt/python-machine-learning-book/pull/13 https://github.com/rasbt/python-machine-learning-book/pull/1... (trying not to change the author's original content much).
- paulsutter 10y agoDeep learning is learning-from-learning (stacking machine learning). And it works because of backprop. That's the basic difference.