12 ms·

The Difference Between AI, Machine Learning, and Deep Learning

288 points by kercker 10y ago

ragebol 10y ago
For machine learning and deep learning based assistive functions, like Google Now, Cortana and Siri, we let the companies behind them gather a lot of our personal data into "Big Data", because then you can do all the statistics. I think a (more) true AI would not need all this data and just be a _personal_ assistant, not needing all the big data of other people too. Maybe initially, once, and then be a good personal assistant, learning to know you like a human personal assistant. It would need some common sense, which is now lacking, mostly.
danvayn 10y ago
Why the hell is the text not black? It's very awful to read on mobile..
harlowja 10y ago
Has anyone figured out how to debug and/or analyze deep ANN? That's the problem I always had, you may get them into a trained state, but good luck figuring out any reason 'why' they ended up in that state (or even what that state really is).
- nightski 10y ago
  IMO that's not really the point of deep learning. I'm not saying no efforts have been made to work towards better understanding, but the types of problems deep learning is good at I do not believe have easily explainable answers. There are machine learning methods out there that are much better at explaining why, but fail hard on problems that DNN is good at. IMHO, this is the nature of the problem, not the solution.
- dave_sullivan 10y ago
  > That's the problem I always had, you may get them into a trained state, but good luck figuring out any reason 'why' they ended up in that state (or even what that state really is). Can you give a specific example of what you mean? I ask because I see this sentiment often, but primarily from people who are very new to deep learning. You can definitely debug a neural network. You mostly want to look at metrics from training such as gradient norms or try adjusting parameters to see if you can get a gain in overall performance as measured by cross validation performance. You can definitely analyze a neural network. You do so by forming hypotheses, preparing datasets that reflect those hypotheses, and running them through your trained models, noting cross validation performance. It's also possible to visualize the weights in various ways, there are many papers written about it. So what do you mean exactly when you say no one has figured out how to debug or analyze DNNs?
  - maaaats 10y ago
    You cannot say: "This image was classified as a stop sign, because this part recognized the shape and this part recognized the color, and this part the text", which you could do with other approaches. When it doesn't discover that it's a stop sign, how do you debug it? Did it recognize the shape.. who knows?
    - augustt 10y ago
      It's definitely possible to get insight into how a CNN would classify something like a stop sign. This paper does a good job of showing how CNNs learn a hierarchy of increasingly complex features to classify images: http://arxiv.org/abs/1311.2901 http://arxiv.org/abs/1311.2901
    - vidarh 10y ago
      > When it doesn't discover that it's a stop sign, how do you debug it? Did it recognize the shape.. who knows? Barring other analytic tools (like looking at which parts contribute the most to the wrong result), the same way you test other things when you have a (somewhat) black box: Form hypotheses and test them. Maybe it didn't recognise the shape, so try adjusting the image to clean it up, and once you have one it recognises, try to reduce and alter the difference between them. Maybe it turns out the image e.g. has the stop sign slightly covered, making the shape look wrong, and there's nothing in the training set like that. Maybe the hue or brightness is off and the training set is mostly all lit a certain way. Test it by adjusting hue and brightness of the test image and see if it gets recognised. And so on. There are plenty of other areas where we are similarly constrained from taking apart that which we're observing, so it's not like this isn't something scientists are dealing with all the time. Within comp.sci. we're just spoiled in that so much of what we do can be easily instrumented, isolated and tested in ways that often lets us determine clear, specific root causes through analysis.
- the8472 10y ago
  That's what deep dream was about to some extent. They tweak the input to maximize the response of specific neurons somewhere in the middle of the network to figure out what those neurons actually "learned".
- nshm 10y ago
  LIME is a software to analyze any machine learning classifier https://github.com/marcotcr/lime https://github.com/marcotcr/lime, they have very good paper in KDD too http://www.kdd.org/kdd2016/papers/files/rfp0573-ribeiroA.pdf http://www.kdd.org/kdd2016/papers/files/rfp0573-ribeiroA.pdf
oliwaw 10y ago
Deep learning is just a rebranding of "neural networks". When neural nets became unpopular in the 90s and early 2000s, people talked about "multilayer networks" (dropping the "neural") since it wasn't really useful to think about this approach from the neuro perspective (since it's such a cartoonish model of real neural networks anyway). Now that very deep networks have become possible, and various graphical models and Bayesian approaches have also been folded under "deep learning" (for example, using back-propagation to learn complicated posterior distributions in variational Bayes) deep learning is not just about vanilla feedforward nets.
- cinquemb 10y ago
  >…since it wasn't really useful from the neuro perspective (since it's such a cartoonish model of real neural networks anyway). Still isn't and still masses of people go on to think that these "neural" networks work the same as neurons in a body do… while neuroscientists are still trying to understand how real neuronal networks operate with a bunch of pet phenomenological theories that most pretty much ignore physics despite the tools used lol
- maaaats 10y ago
  Yeah, I've always heard that expert systems are good, because there you can reason about the solution. For instance diagnosing people based on rules contributed by doctors. You can trace the steps the algorithm takes easily. But for a neural net, you cannot say why this particular net should be trusted, as you don't know how it arrives at a solution. Therefore it's "scary" to use. While I don't agree, it explains why it has been unpopular.
  - joakleaf 10y ago
    Random Decision Forests provide sort of a middle ground. For training, a large set of decision trees are built randomly based on the input features. When classifying input for one tree, each node considers feature value of the input, and decides on a branch. Leafs corresponds to a classification, so when a leaf is reached, the tree has classified the given input. By having a large set of trees, and picking e.g. the most common resulting class (majority vote), we increase accuracy. However, each individual tree can actually be reasoned about. E.g. you can see the analysis (nodes) leading to each class (leafs). I've had some success with RDFs in the past, and highly recommend them! They are very easy to implement, very efficient to train and query, and they seem to work really great on classification of "discrete input" (i.e. where the input feature values are binary or from relatively small sets).
paulsutter 10y ago
I disagree. Machine learning is more general than AI and therefore should be the outer circle (unless you believe the meme "did a regression - called it AI") Deep learning is, yes, a subcase of machine learning, and AI may be a circle within machine learning and enclose deep learning. But truth be told, we will all regret the way we use the term AI now. Eventually the term AI will refer only to general intelligence (aka AGI).
- NhanH 10y ago
  Machine learning and AI are two overlapped fields, it's hard to say one completely encompasses the other. So two overlapped circle would be better?
- paradite 10y ago
  I think the author's illustration is technically correct, given the currently widely adopted definition of AI and its subfields. However, you do have a point that simple machine learning aka linear regression is not really "Artificial Intelligence" if you raise the expectation of "Artificial Intelligence".
- cam_l 10y ago
  Eventually.. eventually, we will drop the A too.
- argonaut 10y ago
  "AI" is a specific term used in academia / research. In that context, ML is most certainly a subset of AI. The fact that many laymen have a different conception of the term is hardly a reason to change all our uses of the term.
- denzil_correa 10y ago
  > Machine learning is more general than AI and therefore should be the outer circle Machine learning is one of the but not the ONLY way to achieve AI. Artificial Intelligence is the objective, machine learning is one way to achieve it.
ktta 10y ago
Also, an interesting read related to this would 'the AI effect'[0]. A lot of the stuff Deep Learning/Machine Learning is able to do today would be looked at as something that only 'true' AI (whatever consensus on what that means is; I think of it as AGI) would be able to do. But as soon as we are able to solve a problem that we think (feel?) only true AI (AGI) would be able to solve, as soon as we know how it was solved, it is no longer a mystery that warrants amazement and we argue that it is not real intelligence, just like the link below states. But just a decade ago, if you saw that a computer was able to recognize pictures better than humans, you would think there was something fishy going on, or we have achieved true AI. My opinion is true AI is which is comparable to human intelligence, in that it is sentient and/or capable of abstract thought, not necessarily being able to hold a conversation or solve concrete mathematics problems, or dump out a story by neural networks. [0]https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect
- argonaut 10y ago
  I hear this repeated a lot. But I've never actually seen a source from, say, the 90's or the 2000s, actually saying something like "if we're able to classify images better than humans, that's true AI." I suspect it's just hearsay that's been repeated over and over again.
  - svantana 10y ago
    There used to be a concept of "AI completeness" which meant that some problems could only be solved by True AI, which would also be able to solve all other human-level problems. Doug Hofstadter writes in Gödel-Escher-Bach that he believes grandmaster-level chess is AI complete. That was obviously false in retrospect, and it's becoming increasingly obvious that there is no AI completeness at all.
    - ragebol 10y ago
      For now, I take this blogpost [1] of Andrej Karpathy as a nice measure for AI hardness. [1] http://karpathy.github.io/2012/10/22/state-of-computer-vision/ http://karpathy.github.io/2012/10/22/state-of-computer-visio...
Azuolas 10y ago
this is a very interesting read. Thank you.
thallukrish 10y ago
If I need to show 1 million cat images to train a neural net to see cats, I wonder how a human brain can figure out cats of any kind just by seeing one or two. Is there something fundamental we are missing in going about building these deep learning stuff ?
- bomdo 10y ago
  Great question. Even a human brain has to train for ~4-5 months to become interested in shapes (https://en.wikipedia.org/wiki/Infant_visual_development https://en.wikipedia.org/wiki/Infant_visual_development). Once the human brain has been trained for these basic shapes for a while, it is able to quickly break down a new class (i.e. a cat) and recognize similar patterns in other images. This is something that is very similar to the way that training a deep NN works. Also don't forget that the current NN are being trained mainly for photos, not moving images. A brain may recognize a cat by its tail-wagging or fur movements, which is a dimension that is completely missing from still images. Also check out this similar post and discussion: https://news.ycombinator.com/item?id=9247851 https://news.ycombinator.com/item?id=9247851
  - vidarh 10y ago
    > Even a human brain has to train for ~4-5 months to become interested in shapes Everyone who wonder how (on a superficial level) grown up humans are so good at learning new categories really should spend time around babies and toddlers and children for this reason... You quickly realise how much training and brain development it actually takes before we're capable of doing much.
    - bpizzi 10y ago
      By far the greatest part of the time I spent with my two toddlers (5 and 3) is spent repeating the same things, again and again, and trying to find shorcuts for learning (which shorcuts might just not exist, afaik). Then one day they start to get it (like "fire burns"), but it's still not there for sure until they experiment it deeply multiple times. The dev in me can't help but see this two little humans as big mighty neural networks who spend their full uptime constantly ingesting tremendous amount of data and being restlessly tuned back by adults and experience :)
bpesquet 10y ago
Great writeup about how deep learning came to be: http://www.andreykurenkov.com/writing/a-brief-history-of-neural-nets-and-deep-learning/ http://www.andreykurenkov.com/writing/a-brief-history-of-neu... (posted here a few months ago).
stefanv 10y ago
There is also machine intelligence: http://numenta.com/blog/machine-intelligence-machine-learning-deep-learning-artificial-intelligence.html http://numenta.com/blog/machine-intelligence-machine-learnin...
deleted 10y ago
[deleted]
bogomipz 10y ago
I was confused by this sentence in the last paragraph: "Deep Learning has enabled many practical applications of machine learning and by extension the overall field of AI." Is it not the reverse - machine learning has enabled deep learning? Can someone comment on how the two - machine learning and Deep learning relate? Is the relationship sequential i.e a data set from machine learning is the the input for a neural network? The diagram had the effect of confusing me.
- qntty 10y ago
  the relationship suggested by the article is that deep learning is a subset of machine learning
  - bogomipz 10y ago
    Sure, that was kind of my point, and as such I would think the sentence should instead read "Machine Learning has enabled many practical applications of deep learning .." No?
unabst 10y ago
There's that one iconic image of neurons suspended in space with bolts of electricity flashing between them. We are told that's how our brains work. We are then shown a diagram by a computer scientist. Instead of cells and thunder, we see circles and arrows. Then we are told there is an algorithm that simulates what the brain does. Viola, we have our artificial neural network. Not only do they look similar, they have two words in common, neural and network! And so for most of us, there is only one logical conclusion: It does what our brain does, so once our computers have the power our brains do, we'll have the singularity! Of course, now we know this is complete bullshit. Basically, computer scientists just took the names and those initial abstractions and ran with it. They never looked back at the biology or how brains actually work. The result is a ton of great research, but they've strayed further and further from neuroscience and from humans. Which is obvious, because they're staring at code and computers all day, not brain meat. If there is one thing AlphaGo proved it is that we've made a ton of progress in computation, but that it's a different direction. Just the fact that average people generally suck at Go should be enough to show that AlphaGo is not human (in many ways it's beyond human). In the meantime, our neuroscientist have made progress also, except, they've done it staring at the actual brain. And now it's to the point where our brains look nothing like that original image our computer scientists were inspired with. Now there is this (Harvard research): https://www.youtube.com/watch?v=8YM7-Od9Wr8 https://www.youtube.com/watch?v=8YM7-Od9Wr8 And this (MIT research): https://www.ted.com/talks/sebastian_seung?language=en https://www.ted.com/talks/sebastian_seung?language=en With advancement comes new vocabulary, and the new word this time is connectome. Some incredibly smart computer scientists will, again, take the term and all the diagrams, and start programming based on it. The result will be Artifical Connectomes, and they will blow our socks off. Now, don't get me wrong. I am not trying to be sarcastic here. This is what _should_ happen. And with every iteration, we will get closer to AGI. It's just that whenever I see articles about machine learning and neural networks, I can't help but think of that classic artist's rendition of neurons firing, and how it's basically complete bullshit. Like Bohr's atom, it's an illustration based on a theory, not reality. Now we have wave function diagrams and connectomes. But as a physicist would tell you, anyone caught with a Bohr's atom is stuck in the 20th century.
- yvdriess 10y ago
threepipeproblm 10y ago
I recently learned that Alexander Stepanov, designer of the C++ STL, has stated that, "I think that object orientedness is almost as much of a hoax as Artificial Intelligence."
ihaveahadron 10y ago
IMO: With so many nonsensical things being in the article, it still seems like we already have AI -- making organisms that grow into brains (humans). And this so called AI still seems like ever more complex machinery. If the computers are learning, I doubt its from a vacuum, its from things (language, etc.), created by humans. I guess my point here is that AI suggests that the computers are "alive" conscious "entities". I don't see it.
vonnik 10y ago
This intro is pretty good, but may be a bit high level for some readers of HN. It seems like it's written for non-technical readers. The idea is that AI, ML and DL are nested like Russian dolls, each subsequent one a subset of the other. It might be better to explain why deep learning is so effective, in clear language: * Deep artificial neural networks are old, relatively simple combinations of math and code that are now able to produce accurate models through brute force because we have 1) vastly more computational power thanks to NVIDIA and distributed run-times; 2) much more data, and much larger labeled datasets thanks to people like Fei-Fei Li at Stanford; 3) better algorithms thanks to the work of Hinton, LeCun, Bengio, Ng, Schmidhuber and a raft of others. Deep is a technical term. It refers to the number of layers through which data passes in a neural net; that is, the number of mathematical operations it is subjected to, and the number of times it is recombined with other inputs. This recombination of inputs, moving deeper into the net, is the basis of feature hierarchy, which is another way of saying: we can cluster and classify data using more complex and abstract representations. That clustering and classification is at the heart of what deep learning does. Another way to think about it is as machine perception. So the overarching narrative in AI is that we've moved from the symbolic rules engines of the chess victors to the interpretation of complex sensory information. For a long time, people would say AI could beat a 30-year-old at chess but couldn't beat a 3-year old at basic tasks. That's no longer true. We can go around beating 3-year-olds at name games all day. AI mind, beginner's mind. But it's important to note that deep learning actually refers to other algorithms besides artificial neural networks. Deep reinforcement learning is one example. RL is also an old set of algorithms, which are goal-oriented. RL helps agents choose the right action in a given state to maximize rewards from the environment. Basically, they learn the function that converts actions to rewards given certain conditions, and that function is non-differentiable; that is, you can't learn it simply by backpropagating error, the way neural nets do. Deep RL is important because the most amazing algorithms, like AlphaGo, are combining deep neural nets (recognize the state of the Go board) with RL (pick the move most likely to succeed) and other components like Monte Carlo Decision Trees (limit the state space we explore). So we're moving beyond perception to algorithms that can make strategic decisions in increasingly complex environments. We've written more about this, and implemented many of these algorithms: http://deeplearning4j.org/ai-machinelearning-deeplearning.html http://deeplearning4j.org/ai-machinelearning-deeplearning.ht... http://deeplearning4j.org/reinforcementlearning.html http://deeplearning4j.org/reinforcementlearning.html http://github.com/deeplearning4j/rl4j http://github.com/deeplearning4j/rl4j

288 points by kercker 10y ago

For machine learning and deep learning based assistive functions, like Google Now, Cortana and Siri, we let the companies behind them gather a lot of our personal data into "Big Data", because then you can do all the statistics. I think a (more) true AI would not need all this data and just be a _personal_ assistant, not needing all the big data of other people too. Maybe initially, once, and then be a good personal assistant, learning to know you like a human personal assistant. It would need some common sense, which is now lacking, mostly.

danvayn 10y ago

Why the hell is the text not black? It's very awful to read on mobile..

harlowja 10y ago

Has anyone figured out how to debug and/or analyze deep ANN? That's the problem I always had, you may get them into a trained state, but good luck figuring out any reason 'why' they ended up in that state (or even what that state really is).

nightski 10y ago
IMO that's not really the point of deep learning. I'm not saying no efforts have been made to work towards better understanding, but the types of problems deep learning is good at I do not believe have easily explainable answers. There are machine learning methods out there that are much better at explaining why, but fail hard on problems that DNN is good at. IMHO, this is the nature of the problem, not the solution.
dave_sullivan 10y ago
> That's the problem I always had, you may get them into a trained state, but good luck figuring out any reason 'why' they ended up in that state (or even what that state really is). Can you give a specific example of what you mean? I ask because I see this sentiment often, but primarily from people who are very new to deep learning. You can definitely debug a neural network. You mostly want to look at metrics from training such as gradient norms or try adjusting parameters to see if you can get a gain in overall performance as measured by cross validation performance. You can definitely analyze a neural network. You do so by forming hypotheses, preparing datasets that reflect those hypotheses, and running them through your trained models, noting cross validation performance. It's also possible to visualize the weights in various ways, there are many papers written about it. So what do you mean exactly when you say no one has figured out how to debug or analyze DNNs?
- maaaats 10y ago
  You cannot say: "This image was classified as a stop sign, because this part recognized the shape and this part recognized the color, and this part the text", which you could do with other approaches. When it doesn't discover that it's a stop sign, how do you debug it? Did it recognize the shape.. who knows?
  - augustt 10y ago
    It's definitely possible to get insight into how a CNN would classify something like a stop sign. This paper does a good job of showing how CNNs learn a hierarchy of increasingly complex features to classify images: http://arxiv.org/abs/1311.2901 http://arxiv.org/abs/1311.2901
  - vidarh 10y ago
    > When it doesn't discover that it's a stop sign, how do you debug it? Did it recognize the shape.. who knows? Barring other analytic tools (like looking at which parts contribute the most to the wrong result), the same way you test other things when you have a (somewhat) black box: Form hypotheses and test them. Maybe it didn't recognise the shape, so try adjusting the image to clean it up, and once you have one it recognises, try to reduce and alter the difference between them. Maybe it turns out the image e.g. has the stop sign slightly covered, making the shape look wrong, and there's nothing in the training set like that. Maybe the hue or brightness is off and the training set is mostly all lit a certain way. Test it by adjusting hue and brightness of the test image and see if it gets recognised. And so on. There are plenty of other areas where we are similarly constrained from taking apart that which we're observing, so it's not like this isn't something scientists are dealing with all the time. Within comp.sci. we're just spoiled in that so much of what we do can be easily instrumented, isolated and tested in ways that often lets us determine clear, specific root causes through analysis.
the8472 10y ago
That's what deep dream was about to some extent. They tweak the input to maximize the response of specific neurons somewhere in the middle of the network to figure out what those neurons actually "learned".
nshm 10y ago
LIME is a software to analyze any machine learning classifier https://github.com/marcotcr/lime https://github.com/marcotcr/lime, they have very good paper in KDD too http://www.kdd.org/kdd2016/papers/files/rfp0573-ribeiroA.pdf http://www.kdd.org/kdd2016/papers/files/rfp0573-ribeiroA.pdf

oliwaw 10y ago

Deep learning is just a rebranding of "neural networks". When neural nets became unpopular in the 90s and early 2000s, people talked about "multilayer networks" (dropping the "neural") since it wasn't really useful to think about this approach from the neuro perspective (since it's such a cartoonish model of real neural networks anyway). Now that very deep networks have become possible, and various graphical models and Bayesian approaches have also been folded under "deep learning" (for example, using back-propagation to learn complicated posterior distributions in variational Bayes) deep learning is not just about vanilla feedforward nets.

cinquemb 10y ago
>…since it wasn't really useful from the neuro perspective (since it's such a cartoonish model of real neural networks anyway). Still isn't and still masses of people go on to think that these "neural" networks work the same as neurons in a body do… while neuroscientists are still trying to understand how real neuronal networks operate with a bunch of pet phenomenological theories that most pretty much ignore physics despite the tools used lol
maaaats 10y ago
Yeah, I've always heard that expert systems are good, because there you can reason about the solution. For instance diagnosing people based on rules contributed by doctors. You can trace the steps the algorithm takes easily. But for a neural net, you cannot say why this particular net should be trusted, as you don't know how it arrives at a solution. Therefore it's "scary" to use. While I don't agree, it explains why it has been unpopular.
- joakleaf 10y ago
  Random Decision Forests provide sort of a middle ground. For training, a large set of decision trees are built randomly based on the input features. When classifying input for one tree, each node considers feature value of the input, and decides on a branch. Leafs corresponds to a classification, so when a leaf is reached, the tree has classified the given input. By having a large set of trees, and picking e.g. the most common resulting class (majority vote), we increase accuracy. However, each individual tree can actually be reasoned about. E.g. you can see the analysis (nodes) leading to each class (leafs). I've had some success with RDFs in the past, and highly recommend them! They are very easy to implement, very efficient to train and query, and they seem to work really great on classification of "discrete input" (i.e. where the input feature values are binary or from relatively small sets).

paulsutter 10y ago

I disagree. Machine learning is more general than AI and therefore should be the outer circle (unless you believe the meme "did a regression - called it AI") Deep learning is, yes, a subcase of machine learning, and AI may be a circle within machine learning and enclose deep learning. But truth be told, we will all regret the way we use the term AI now. Eventually the term AI will refer only to general intelligence (aka AGI).

NhanH 10y ago
Machine learning and AI are two overlapped fields, it's hard to say one completely encompasses the other. So two overlapped circle would be better?
paradite 10y ago
I think the author's illustration is technically correct, given the currently widely adopted definition of AI and its subfields. However, you do have a point that simple machine learning aka linear regression is not really "Artificial Intelligence" if you raise the expectation of "Artificial Intelligence".
cam_l 10y ago
Eventually.. eventually, we will drop the A too.
argonaut 10y ago
"AI" is a specific term used in academia / research. In that context, ML is most certainly a subset of AI. The fact that many laymen have a different conception of the term is hardly a reason to change all our uses of the term.
denzil_correa 10y ago
> Machine learning is more general than AI and therefore should be the outer circle Machine learning is one of the but not the ONLY way to achieve AI. Artificial Intelligence is the objective, machine learning is one way to achieve it.

ktta 10y ago

Also, an interesting read related to this would 'the AI effect'[0]. A lot of the stuff Deep Learning/Machine Learning is able to do today would be looked at as something that only 'true' AI (whatever consensus on what that means is; I think of it as AGI) would be able to do. But as soon as we are able to solve a problem that we think (feel?) only true AI (AGI) would be able to solve, as soon as we know how it was solved, it is no longer a mystery that warrants amazement and we argue that it is not real intelligence, just like the link below states. But just a decade ago, if you saw that a computer was able to recognize pictures better than humans, you would think there was something fishy going on, or we have achieved true AI. My opinion is true AI is which is comparable to human intelligence, in that it is sentient and/or capable of abstract thought, not necessarily being able to hold a conversation or solve concrete mathematics problems, or dump out a story by neural networks. [0]https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect

argonaut 10y ago
I hear this repeated a lot. But I've never actually seen a source from, say, the 90's or the 2000s, actually saying something like "if we're able to classify images better than humans, that's true AI." I suspect it's just hearsay that's been repeated over and over again.
- svantana 10y ago
  There used to be a concept of "AI completeness" which meant that some problems could only be solved by True AI, which would also be able to solve all other human-level problems. Doug Hofstadter writes in Gödel-Escher-Bach that he believes grandmaster-level chess is AI complete. That was obviously false in retrospect, and it's becoming increasingly obvious that there is no AI completeness at all.
  - ragebol 10y ago
    For now, I take this blogpost [1] of Andrej Karpathy as a nice measure for AI hardness. [1] http://karpathy.github.io/2012/10/22/state-of-computer-vision/ http://karpathy.github.io/2012/10/22/state-of-computer-visio...

Azuolas 10y ago

this is a very interesting read. Thank you.

thallukrish 10y ago

If I need to show 1 million cat images to train a neural net to see cats, I wonder how a human brain can figure out cats of any kind just by seeing one or two. Is there something fundamental we are missing in going about building these deep learning stuff ?

bomdo 10y ago
Great question. Even a human brain has to train for ~4-5 months to become interested in shapes (https://en.wikipedia.org/wiki/Infant_visual_development https://en.wikipedia.org/wiki/Infant_visual_development). Once the human brain has been trained for these basic shapes for a while, it is able to quickly break down a new class (i.e. a cat) and recognize similar patterns in other images. This is something that is very similar to the way that training a deep NN works. Also don't forget that the current NN are being trained mainly for photos, not moving images. A brain may recognize a cat by its tail-wagging or fur movements, which is a dimension that is completely missing from still images. Also check out this similar post and discussion: https://news.ycombinator.com/item?id=9247851 https://news.ycombinator.com/item?id=9247851
- vidarh 10y ago
  > Even a human brain has to train for ~4-5 months to become interested in shapes Everyone who wonder how (on a superficial level) grown up humans are so good at learning new categories really should spend time around babies and toddlers and children for this reason... You quickly realise how much training and brain development it actually takes before we're capable of doing much.
  - bpizzi 10y ago
    By far the greatest part of the time I spent with my two toddlers (5 and 3) is spent repeating the same things, again and again, and trying to find shorcuts for learning (which shorcuts might just not exist, afaik). Then one day they start to get it (like "fire burns"), but it's still not there for sure until they experiment it deeply multiple times. The dev in me can't help but see this two little humans as big mighty neural networks who spend their full uptime constantly ingesting tremendous amount of data and being restlessly tuned back by adults and experience :)

bpesquet 10y ago

Great writeup about how deep learning came to be: http://www.andreykurenkov.com/writing/a-brief-history-of-neural-nets-and-deep-learning/ http://www.andreykurenkov.com/writing/a-brief-history-of-neu... (posted here a few months ago).

stefanv 10y ago

There is also machine intelligence: http://numenta.com/blog/machine-intelligence-machine-learning-deep-learning-artificial-intelligence.html http://numenta.com/blog/machine-intelligence-machine-learnin...

deleted 10y ago

[deleted]

bogomipz 10y ago

I was confused by this sentence in the last paragraph: "Deep Learning has enabled many practical applications of machine learning and by extension the overall field of AI." Is it not the reverse - machine learning has enabled deep learning? Can someone comment on how the two - machine learning and Deep learning relate? Is the relationship sequential i.e a data set from machine learning is the the input for a neural network? The diagram had the effect of confusing me.

qntty 10y ago
the relationship suggested by the article is that deep learning is a subset of machine learning
- bogomipz 10y ago
  Sure, that was kind of my point, and as such I would think the sentence should instead read "Machine Learning has enabled many practical applications of deep learning .." No?

unabst 10y ago

There's that one iconic image of neurons suspended in space with bolts of electricity flashing between them. We are told that's how our brains work. We are then shown a diagram by a computer scientist. Instead of cells and thunder, we see circles and arrows. Then we are told there is an algorithm that simulates what the brain does. Viola, we have our artificial neural network. Not only do they look similar, they have two words in common, neural and network! And so for most of us, there is only one logical conclusion: It does what our brain does, so once our computers have the power our brains do, we'll have the singularity! Of course, now we know this is complete bullshit. Basically, computer scientists just took the names and those initial abstractions and ran with it. They never looked back at the biology or how brains actually work. The result is a ton of great research, but they've strayed further and further from neuroscience and from humans. Which is obvious, because they're staring at code and computers all day, not brain meat. If there is one thing AlphaGo proved it is that we've made a ton of progress in computation, but that it's a different direction. Just the fact that average people generally suck at Go should be enough to show that AlphaGo is not human (in many ways it's beyond human). In the meantime, our neuroscientist have made progress also, except, they've done it staring at the actual brain. And now it's to the point where our brains look nothing like that original image our computer scientists were inspired with. Now there is this (Harvard research): https://www.youtube.com/watch?v=8YM7-Od9Wr8 https://www.youtube.com/watch?v=8YM7-Od9Wr8 And this (MIT research): https://www.ted.com/talks/sebastian_seung?language=en https://www.ted.com/talks/sebastian_seung?language=en With advancement comes new vocabulary, and the new word this time is connectome. Some incredibly smart computer scientists will, again, take the term and all the diagrams, and start programming based on it. The result will be Artifical Connectomes, and they will blow our socks off. Now, don't get me wrong. I am not trying to be sarcastic here. This is what _should_ happen. And with every iteration, we will get closer to AGI. It's just that whenever I see articles about machine learning and neural networks, I can't help but think of that classic artist's rendition of neurons firing, and how it's basically complete bullshit. Like Bohr's atom, it's an illustration based on a theory, not reality. Now we have wave function diagrams and connectomes. But as a physicist would tell you, anyone caught with a Bohr's atom is stuck in the 20th century.

yvdriess 10y ago

threepipeproblm 10y ago

I recently learned that Alexander Stepanov, designer of the C++ STL, has stated that, "I think that object orientedness is almost as much of a hoax as Artificial Intelligence."

ihaveahadron 10y ago

IMO: With so many nonsensical things being in the article, it still seems like we already have AI -- making organisms that grow into brains (humans). And this so called AI still seems like ever more complex machinery. If the computers are learning, I doubt its from a vacuum, its from things (language, etc.), created by humans. I guess my point here is that AI suggests that the computers are "alive" conscious "entities". I don't see it.

vonnik 10y ago

This intro is pretty good, but may be a bit high level for some readers of HN. It seems like it's written for non-technical readers. The idea is that AI, ML and DL are nested like Russian dolls, each subsequent one a subset of the other. It might be better to explain why deep learning is so effective, in clear language: * Deep artificial neural networks are old, relatively simple combinations of math and code that are now able to produce accurate models through brute force because we have 1) vastly more computational power thanks to NVIDIA and distributed run-times; 2) much more data, and much larger labeled datasets thanks to people like Fei-Fei Li at Stanford; 3) better algorithms thanks to the work of Hinton, LeCun, Bengio, Ng, Schmidhuber and a raft of others. Deep is a technical term. It refers to the number of layers through which data passes in a neural net; that is, the number of mathematical operations it is subjected to, and the number of times it is recombined with other inputs. This recombination of inputs, moving deeper into the net, is the basis of feature hierarchy, which is another way of saying: we can cluster and classify data using more complex and abstract representations. That clustering and classification is at the heart of what deep learning does. Another way to think about it is as machine perception. So the overarching narrative in AI is that we've moved from the symbolic rules engines of the chess victors to the interpretation of complex sensory information. For a long time, people would say AI could beat a 30-year-old at chess but couldn't beat a 3-year old at basic tasks. That's no longer true. We can go around beating 3-year-olds at name games all day. AI mind, beginner's mind. But it's important to note that deep learning actually refers to other algorithms besides artificial neural networks. Deep reinforcement learning is one example. RL is also an old set of algorithms, which are goal-oriented. RL helps agents choose the right action in a given state to maximize rewards from the environment. Basically, they learn the function that converts actions to rewards given certain conditions, and that function is non-differentiable; that is, you can't learn it simply by backpropagating error, the way neural nets do. Deep RL is important because the most amazing algorithms, like AlphaGo, are combining deep neural nets (recognize the state of the Go board) with RL (pick the move most likely to succeed) and other components like Monte Carlo Decision Trees (limit the state space we explore). So we're moving beyond perception to algorithms that can make strategic decisions in increasingly complex environments. We've written more about this, and implemented many of these algorithms: http://deeplearning4j.org/ai-machinelearning-deeplearning.html http://deeplearning4j.org/ai-machinelearning-deeplearning.ht... http://deeplearning4j.org/reinforcementlearning.html http://deeplearning4j.org/reinforcementlearning.html http://github.com/deeplearning4j/rl4j http://github.com/deeplearning4j/rl4j