3 ms·

I'm Jeremy Howard, one of the people quoted in the article (and the one in the crazy photo!) AMA about applications of deep learning. My views from 18 months ag

by jph00 11y ago

I'm Jeremy Howard, one of the people quoted in the article (and the one in the crazy photo!) AMA about applications of deep learning. My views from 18 months ago are in this talk on TED.com: http://www.ted.com/talks/jeremy_howard_the_wonderful_and_terrifying_implications_of_computers_that_can_learn http://www.ted.com/talks/jeremy_howard_the_wonderful_and_ter... . In the time since then much more of the potential of deep learning has been realized, but we're still in the very earliest days - I'd say something like the internet, prior to the development of the first commercial ISPs or the web. Just like today we don't really talk about "internet companies" (since the internet is just a tool that we use, not an end in itself), you also should not expect to see "deep learning companies" be successful as an end in itself. Rather, some companies will use deep learning as a key tool to fix key problems in current processes. For example, http://www.enlitic.com http://www.enlitic.com , my company, is first and foremost a medical company, and uses deep learning as a tool to fix some very specific problems in patient care. PS: I greatly dislike the term "artificial intelligence". It's as meaningless and distracting as that other horrible phrase, "big data". I have no idea whether deep learning algorithms could lead to "intelligence", or what that even means. What matters is what a technology can do , and in this case, deep learning can do a great many interesting things.

rhema 11y ago
Hi Jeremy, As I understand it, there is a big difference between "strong" (e.g. the terminator) and "weak" (e.g. automatic image tagging) AI. Are these basically philosophical terms by now? The knowledge engines of Watson and Wolfram Alpha are still one-step and data driven. Is there any hope for strong AI, or is it mostly a pipe dream?
- jph00 11y ago
  I don't think we have any reason to believe that strong AI is impossible - in fact, I can't see how it could be impossible (since we have an existence proof, in ourselves, that it's possible!) But I don't see any way to know whether we are making progress towards strong AI, or whether deep learning could be a foundation for it. The biological connections are certainly becoming more clear, but it still seems like very early days for this kind of conclusion.
- deleted 11y ago
  [deleted]
PeCaN 11y ago
What do you think of alternative, non-neural-network approaches to deep learning such as Bayesian networks[1] or Support Vector Machines (which as I understand it are largely dead, but the theory behind them is interesting and could lead to more developments). Is the future of deep learning just massive neural networks on huge GPU clusters? Is anyone big doing research on new approaches, or just focusing on fixing the problems with neural networks? (Which, given that the whole idea of deep NNs is rather new anyway, I suppose it might be a little early to look at other algorithms.) 1. http://www.psy.cmu.edu/~ckemp/papers/maask09.pdf http://www.psy.cmu.edu/~ckemp/papers/maask09.pdf
- jimbokun 11y ago
  "Support Vector Machines (which as I understand it are largely dead, but the theory behind them is interesting and could lead to more developments)." Wow, that happened fast. SVMs seemed all the rage a few years ago. I think it shows the importance of understanding the formalisms underlying machine learning approaches, and their corresponding strengths and weaknesses. Neural networks had been on the outs for a long time, before the realization that having much more data to train against could massively increase their usefulness. Wonder what the next ML technique to see a resurgence will be.
- jph00 11y ago
  The difference with deep neural networks is that they can do so much of their own feature generation. It's very different to bayesian networks or SVMs. Because they can create their own features (to a large extent) they can handle unstructured data more simply and more accurately. Neural nets are dramatically improving every month. Recently, the major developments have been around using a wider variety of differentiable functions as layers in the network - see for example Spatial Transformer Networks, and also the more general architectures such as Neural Turing Machines.
karmacondon 11y ago
You compared the current state of deep learning to the early internet. Can you clarify that in any way? I'm having a hard time seeing how deep learning could improve exponentially from where it is now.
- jph00 11y ago
  Sure. Deep learning is a tool that can impact a large percentage of processes that we use to create and provide products and services. Many of these processes involve human steps because humans are uniquely able to provide the perception, language interpretation, or other complex pattern recognition tasks. Replacing or augmenting people in these situations with deep learning driven systems would allow many of these processes to become orders of magnitude cheaper, faster, and more accurate. I believe that the internet was a similar technology - a tool that could make many processes much cheaper, faster, and more accurate. So it's not so much about deep learning itself improving exponentially (which I don't have an informed opinion of, because I don't think it's possible to know), but about the drastic changes that will occur as it is built into the products and services that can use it.
simonebrunozzi 11y ago
Hi Jeremy (Simone Brunozzi here), good to see you here on Hacker News. I very much agree with your disliking of "Big Data" - a term that I've see misused almost 100% of the times. On "Artificial Intelligence", I think it serves the purpose of defining a category for the layman, and doesn't necessarily confuse people as much as "Big Data" does.
- jph00 11y ago
  I think "machine learning" is more accurate and useful, personally.
grondilu 11y ago
> I greatly dislike the term "artificial intelligence". It's as meaningless and distracting as that other horrible phrase, "big data". I have no idea whether deep learning algorithms could lead to "intelligence", or what that even means. What matters is what a technology can do , and in this case, deep learning can do a great many interesting things. The term "artificial intelligence" refers to more than what technology can do. It also refers to what technology could or will do. That why this term matters for the public, as it points at potential future issues described in Science-Fiction. As for what it means, I could agree that it's difficult to coin it precisely, but it's not meaningless either. Some concepts have a meaning regardless of our ability to express this meaning accurately with words. It's a case of "I know it when I see it."
andy16 11y ago
Hi Jeremy, thanks for doing another AMA! I recently watched a great video of you performing deep learning in Excel: https://www.youtube.com/watch?v=qnoLMkosHuE https://www.youtube.com/watch?v=qnoLMkosHuE I was particularly amazed after you made a few tweaks to your learning, which drastically reduced the error rate (i.e introduced adagrad, automatic learning rate/annealing, momentum...). I had never heard of these techniques before watching this - what do you find is the best way regarding how to stay up-to-date with the latest machine learning algorithms/improvements? Thanks

by jph00 11y ago

rhema 11y ago

Hi Jeremy, As I understand it, there is a big difference between "strong" (e.g. the terminator) and "weak" (e.g. automatic image tagging) AI. Are these basically philosophical terms by now? The knowledge engines of Watson and Wolfram Alpha are still one-step and data driven. Is there any hope for strong AI, or is it mostly a pipe dream?

jph00 11y ago
I don't think we have any reason to believe that strong AI is impossible - in fact, I can't see how it could be impossible (since we have an existence proof, in ourselves, that it's possible!) But I don't see any way to know whether we are making progress towards strong AI, or whether deep learning could be a foundation for it. The biological connections are certainly becoming more clear, but it still seems like very early days for this kind of conclusion.
deleted 11y ago
[deleted]

PeCaN 11y ago

What do you think of alternative, non-neural-network approaches to deep learning such as Bayesian networks[1] or Support Vector Machines (which as I understand it are largely dead, but the theory behind them is interesting and could lead to more developments). Is the future of deep learning just massive neural networks on huge GPU clusters? Is anyone big doing research on new approaches, or just focusing on fixing the problems with neural networks? (Which, given that the whole idea of deep NNs is rather new anyway, I suppose it might be a little early to look at other algorithms.) 1. http://www.psy.cmu.edu/~ckemp/papers/maask09.pdf http://www.psy.cmu.edu/~ckemp/papers/maask09.pdf

jimbokun 11y ago
"Support Vector Machines (which as I understand it are largely dead, but the theory behind them is interesting and could lead to more developments)." Wow, that happened fast. SVMs seemed all the rage a few years ago. I think it shows the importance of understanding the formalisms underlying machine learning approaches, and their corresponding strengths and weaknesses. Neural networks had been on the outs for a long time, before the realization that having much more data to train against could massively increase their usefulness. Wonder what the next ML technique to see a resurgence will be.
jph00 11y ago
The difference with deep neural networks is that they can do so much of their own feature generation. It's very different to bayesian networks or SVMs. Because they can create their own features (to a large extent) they can handle unstructured data more simply and more accurately. Neural nets are dramatically improving every month. Recently, the major developments have been around using a wider variety of differentiable functions as layers in the network - see for example Spatial Transformer Networks, and also the more general architectures such as Neural Turing Machines.

karmacondon 11y ago

You compared the current state of deep learning to the early internet. Can you clarify that in any way? I'm having a hard time seeing how deep learning could improve exponentially from where it is now.

jph00 11y ago
Sure. Deep learning is a tool that can impact a large percentage of processes that we use to create and provide products and services. Many of these processes involve human steps because humans are uniquely able to provide the perception, language interpretation, or other complex pattern recognition tasks. Replacing or augmenting people in these situations with deep learning driven systems would allow many of these processes to become orders of magnitude cheaper, faster, and more accurate. I believe that the internet was a similar technology - a tool that could make many processes much cheaper, faster, and more accurate. So it's not so much about deep learning itself improving exponentially (which I don't have an informed opinion of, because I don't think it's possible to know), but about the drastic changes that will occur as it is built into the products and services that can use it.

simonebrunozzi 11y ago

Hi Jeremy (Simone Brunozzi here), good to see you here on Hacker News. I very much agree with your disliking of "Big Data" - a term that I've see misused almost 100% of the times. On "Artificial Intelligence", I think it serves the purpose of defining a category for the layman, and doesn't necessarily confuse people as much as "Big Data" does.

jph00 11y ago
I think "machine learning" is more accurate and useful, personally.

grondilu 11y ago

> I greatly dislike the term "artificial intelligence". It's as meaningless and distracting as that other horrible phrase, "big data". I have no idea whether deep learning algorithms could lead to "intelligence", or what that even means. What matters is what a technology can do , and in this case, deep learning can do a great many interesting things. The term "artificial intelligence" refers to more than what technology can do. It also refers to what technology could or will do. That why this term matters for the public, as it points at potential future issues described in Science-Fiction. As for what it means, I could agree that it's difficult to coin it precisely, but it's not meaningless either. Some concepts have a meaning regardless of our ability to express this meaning accurately with words. It's a case of "I know it when I see it."

andy16 11y ago

Hi Jeremy, thanks for doing another AMA! I recently watched a great video of you performing deep learning in Excel: https://www.youtube.com/watch?v=qnoLMkosHuE https://www.youtube.com/watch?v=qnoLMkosHuE I was particularly amazed after you made a few tweaks to your learning, which drastically reduced the error rate (i.e introduced adagrad, automatic learning rate/annealing, momentum...). I had never heard of these techniques before watching this - what do you find is the best way regarding how to stay up-to-date with the latest machine learning algorithms/improvements? Thanks