21 ms·

Deep Learning - The Biggest Data Science Breakthrough of the Decade

127 points by jph00 14y ago

jph00 14y ago
I created this talk for the Enterprise Big Data track of O'Reilly's Strata conference - so it's not a technical description of how deep learning works. Rather, it's an attempt to show why it's important, and how it fits into current data science trends. The "Biggest Data Science Breakthrough of the Decade" in the title is a rather bold claim, I know... But I think it might be justified. If there are are bigger breakthroughs, I'd be interested in people's thoughts about what they might be.
- pyre 14y ago
  If by "of the Decade," you mean 2010-2020, then you've still got another 7 years to be proven right/wrong.
- spikels 14y ago
  Not sure which decade you are talking about. If you mean the 2010s or the next 10 years we'll just have to see what the next 7 or 10 years bring. But if you mean the past 10 years I would have to say that the "distributed storage and processing" revolution (Hadoop and others) has had a much bigger impact on data science than all of neural networks including deep networks. Why the need to hype what is already a well publicized development? I'm starting to cringe whenever I hear "data science" or "big data" and I love this stuff.
patrickk 14y ago
If you're watching the webcast, skip to 3:20 to avoid all the promo and buildup fluff at the start.
daniel-cussen 14y ago
Does anyone know how RAM-intensive deep learning is? If the answer is "not very," I think the GA144 might be a good candidate because it's a lot of CPU-capable (independently-branching) cores.
- lrei 14y ago
  dont have any numbers for you but it's typically CPU-bound and not memory-bound. deep learning is often done using GPUs because of massive-parallelism.
  - wladimir 14y ago
    It may be computation-bound (I'm not sure) but training deep networks generally does use a lot of memory, because of the giant training sets. You're right that GPUs are a good fit, for example libraries such a theano exploit this.
wmat 14y ago
Can this be viewed without registering?
- jph00 14y ago
  I don't think so. I even had to register to view - and it's my talk! I haven't received any marketing stuff from Cloudera or O'Reilly however. Honestly, I doubt those companies would do anything questionable with registrations.
  - learningram 14y ago
    I wish there was a download option
ergodic 14y ago
Well, it is definitely something but it being the "Breakthrough of the Decade" seems pretty unlikely to me (given my available evidence). I do not know well other examples beyond case of Automatic Speech Recognition, but since this case caused a lot of noise, I bet it is responsible for a reasonable chunk of the Deep learning "buzz". Here is my take about this. If you look at papers from Microsoft like Seide et al 2011 and similar papers the reported improvement against state of the art (up to 30%) is really impressive and seems solid. Now, the technique is more or less using a very big multi-layer perceptron (MLP), a technique already established two decades ago (or more). There is some fancy stuff like the deep belief network based initialization, but it does not make big differences. The core of the recipe itself is not very new. What has changed is the scale of data we have available and the size of the models that we can handle. With this I am not implying that this is not a very interesting discovery. But it is important to bear in mind that the change in the amount of data could also make other 20 year old techniques interesting again. On the other hand, neural networks had a bad name in the last years for understandable reasons. They are a blackbox, or at least less transparent than the statistical methods. This makes them prone to cause the "black box delusion" effect. You hear a new algorithm is in town, it has fancy stuff like remotely resembling human thinking architectures or cool math but you can not completely grasp it guts, then "voila!" suddenly you are overestimating its relevance and scope of applicably. MLPs were hailed as "the" tool for machine learning already once, I think for these same reasons. For me the right position here is a prudent skepticism. On the other hand, this should also push people to try new/old radical stuff since the rules of the game seem to be changing, it is not a moment to be conservative in ML research :).
- lrei 14y ago
  I've heard this argument ever since Norvig's Unreasonable Effectiveness of Data. While having a ton of data available is great, it has its limits. I believe you are overestimating the effectiveness of data (as, imo, Norvig did). And here specifically, it's not the case for the hype: from the NYT article [1]: "The achievement was particularly impressive because the team decided to enter the contest at the last minute and designed its software with no specific knowledge about how the molecules bind to their targets. The students were also working with a relatively small set of data; neural nets typically perform well only with very large ones." NNs in general have enjoyed lots of successful practical (commercial) applications in pattern recognition though they were sort of replaced in the "state-of-the-art" by SVMs in many cases until RBMs and DBNs came along. I agree with your caution for skepticism though, only time will tell how good DBNs are. I think the black box criticism is BS for the most part. In some cases (google's search being a famous example) it might be great to have a human readable and tweakable solution (assuming you have the resources) but for something like recognising handwritten digits from images, not so much. [1] http://www.nytimes.com/2012/11/24/science/scientists-see-advances-in-deep-learning-a-part-of-artificial-intelligence.html?_r=0 http://www.nytimes.com/2012/11/24/science/scientists-see-adv...
zby 14y ago
I registered and listened for maybe 9 minutes - and nothing interesting, just talk about the talk. How I hate webcasts!
- jph00 14y ago
  I believe the talk about the talk only covers 3 min 20 s.
- anigbrowl 14y ago
  Agreed. Don't care much for podcasts either, although I can see the value for people who drive frequently.
rm999 14y ago
I'm concerned deep networks are being overhyped. They're certainly exciting, but they haven't seen much use in industry yet; it's too early to make claims about how they have impacted data science. Also, data science involves a lot more than building predictive models. In my experience >95% of effort goes into something other than building a model. In kaggle contests you usually concentrate on that <5%, which IMO is the fun part but it's not the reality of industry. There are many big breakthroughs in data science that don't involve model building. edit: I haven't listened to the podcast yet (at work), my comment is more about the title.
- lrei 14y ago
  I'm genuinely curious (not being snarky or wtv): what do you put 95% of your effort into?
  - mshron 14y ago
    Not the OP but: * Problem definition * Infrastructure * Data transformation * Exploratory analysis (arguably part of model work) * Results presentation Then again, this is an ongoing disagreement I have with the Kaggle folks over what constitutes "data science," where I'm pretty confident that "applied machine learning" is a better explanation of what their contests are about.
    - rm999 14y ago
      I kind of got them to say it here: https://news.ycombinator.com/item?id=4655927 https://news.ycombinator.com/item?id=4655927 BTW, I'm a big fan of the data analysis that came out of okcupid, is that all your work?
    - lrei 14y ago
      I see. Thanks. I'd say data transformation is a part of feature engineering (commonly the bulk of the effort in a ML application). And exploratory analysis is part of model work. W/o those 2 one would be building a model out of dreams and wishes. Data Science is probably a poorly chosen description. I'd say common use includes infrastructure work which for most of us consists in engineering work.
Aloisius 14y ago
I would love to watch this, but O'Reilly's presentation streamer is awful. I tried jumping ahead, but the video stream doesn't actually jump with me so I end up listening to one part and watching another (tried under FireFox, Safari and Chrome on Mac). I don't suppose someone has an alternative version somewhere?
softbuilder 14y ago
This is a new topic to me. Are there any whitepapers or open source projects touching on this?
pbharrin 14y ago
You need Flash to watch this even on Chrome.
hsshah 14y ago
We should discourage submissions like this here on HN that require registration to view the content. So even though this topic is of great interest of mine, I will not be upvoting it. Sorry.
- drucken 14y ago
  While I agree with you, note there is already flagrant "abuse" of this on HN by the posting of newspaper links with paywalls, in particular the New York Times.
stiff 14y ago
I am sorry, but does anyone else have the impression that a lot of people is commenting here with so much confidence while clearly not knowing anything about the topic? It takes almost an equivalent of an undergraduate mathematics degree and than a lot of experience in ML itself to get a decent understanding how things like Deep Belief Networks work, so I don't wonder none of the comments so far hinted at any understanding of anything particular about deep learning, just general derogatory comments "not used in industry", "overhyped", pointers to whatever someone heard in an undergraduate ML class on older types of networks etc. Maybe if you don't have anything on topic to say, just do not comment? You really are not obliged to have an opinion on everything. (Waiting for the downvotes)
- ergodic 14y ago
  rather than being so aggressive and waiting for downvotes you could choose to be constructive ;) and answer a concise technical question to a sub-thread that you yourself started https://news.ycombinator.com/item?id=5377101 https://news.ycombinator.com/item?id=5377101
- spikels 14y ago
  It is purely a matter of opinion whether DBNs are "overhyped" but I hope you would agree that they are currently being "hyped". And I hope you understand how this can actually damage the potential of what is likely some very good technology like say like what happened to neural nets. :) It is also a matter of opinion how widely they are being used in industry. Certainly they are being studied in many companies but they do not a appear to used much in production because of their complexity and high training cost. This is still cutting edge technology. In my experience most professionally trained mathematicians and statisticians are still pretty skeptical of these claims. Wouldn't you agree?
  - stiff 14y ago
    It is OK to be skeptical, all I am trying to say is that most of the comments leave the impression of the poster trying to lean over backwards to say something at all related to the topic, often ending up with generic truisms. For example how is model building being a small part of practical ML criticism of deep learning? How is it on-topic at all?
espeed 14y ago
Quoc V. Le (http://ai.stanford.edu/~quocle/ http://ai.stanford.edu/~quocle/) of Stanford and Google did a talk today at Univ. of Washington: "Scaling deep learning to 10,000 cores and beyond": Presentation Univ. of Washington (March 14, 2013) https://www.cs.washington.edu/htbin-post/mvis/mvis?ID=1338 You can see one of Quoc's previous talks online: "Tera-scale deep learning: - Quoc V. Le from ML Lunch @ CMU http://vimeo.com/52332329 http://vimeo.com/52332329 You may remember Jeff Dean's (http://research.google.com/pubs/jeff.html http://research.google.com/pubs/jeff.html) post on this: https://plus.google.com/118227548810368513262/posts/PozFb134egM https://plus.google.com/118227548810368513262/posts/PozFb134... The corresponding research at Google... "Building high-level features using large scale unsupervised learning" http://research.google.com/pubs/pub38115.html http://research.google.com/pubs/pub38115.html http://research.google.com/archive/unsupervised_icml2012.html http://research.google.com/archive/unsupervised_icml2012.htm... Previous HN discussion: https://news.ycombinator.com/item?id=4145558 https://news.ycombinator.com/item?id=4145558 -- How Many Computers to Identify a Cat? 16,000 http://www.nytimes.com/2012/06/26/technology/in-a-big-network-of-computers-evidence-of-machine-learning.html?_r=2&smid=go-share&#38 http://www.nytimes.com/2012/06/26/technology/in-a-big-networ...;
pilooch 14y ago
Deep learning is so attractive to any AI and machine learning practitioner! The results are beautiful to witness or read about. This is clearly another step in a direction that many of us have been waiting (or working on) for a long time! That said, AI, like every other sciences, experiences trends and bubbles. If you give a decent look to usages and problem solving with machine learning, deep learning techniques are not exactly the final answer. Typically, they're slow to train, to my knowledge there is no good 'online' algorithm yet to train them (i.e. for autoencoders, recursive autoencoders, Boltzman machines). Many applications, and a trend toward 'lifelong learning'[1], require fast incremental learning that yields results in near real-time, or at least in minutes rather than days. I've compared a couple of unsupervised machine learning algorithms with recursive autoencoders: the latter can learn deeper, very often, but at a computational costs (days vs seconds). Deep learning computation will improve, for sure though. [1] http://cs.brynmawr.edu/~eeaton/AAAI-SSS13-LML/ http://cs.brynmawr.edu/~eeaton/AAAI-SSS13-LML/
wfunction 14y ago
Why are they are they tricking us into giving away personal information?
amit_m 14y ago
Argh! Worst streaming experience ever. Also, no contents in the first 20 minutes. It took me a while to understand that the slides were in a popup was blocked. After reloading, the slides don't match the audio.

127 points by jph00 14y ago

jph00 14y ago

I created this talk for the Enterprise Big Data track of O'Reilly's Strata conference - so it's not a technical description of how deep learning works. Rather, it's an attempt to show why it's important, and how it fits into current data science trends. The "Biggest Data Science Breakthrough of the Decade" in the title is a rather bold claim, I know... But I think it might be justified. If there are are bigger breakthroughs, I'd be interested in people's thoughts about what they might be.

pyre 14y ago
If by "of the Decade," you mean 2010-2020, then you've still got another 7 years to be proven right/wrong.
spikels 14y ago
Not sure which decade you are talking about. If you mean the 2010s or the next 10 years we'll just have to see what the next 7 or 10 years bring. But if you mean the past 10 years I would have to say that the "distributed storage and processing" revolution (Hadoop and others) has had a much bigger impact on data science than all of neural networks including deep networks. Why the need to hype what is already a well publicized development? I'm starting to cringe whenever I hear "data science" or "big data" and I love this stuff.

patrickk 14y ago

If you're watching the webcast, skip to 3:20 to avoid all the promo and buildup fluff at the start.

daniel-cussen 14y ago

Does anyone know how RAM-intensive deep learning is? If the answer is "not very," I think the GA144 might be a good candidate because it's a lot of CPU-capable (independently-branching) cores.

lrei 14y ago
dont have any numbers for you but it's typically CPU-bound and not memory-bound. deep learning is often done using GPUs because of massive-parallelism.
- wladimir 14y ago
  It may be computation-bound (I'm not sure) but training deep networks generally does use a lot of memory, because of the giant training sets. You're right that GPUs are a good fit, for example libraries such a theano exploit this.

wmat 14y ago

Can this be viewed without registering?

jph00 14y ago
I don't think so. I even had to register to view - and it's my talk! I haven't received any marketing stuff from Cloudera or O'Reilly however. Honestly, I doubt those companies would do anything questionable with registrations.
- learningram 14y ago
  I wish there was a download option

ergodic 14y ago

Well, it is definitely something but it being the "Breakthrough of the Decade" seems pretty unlikely to me (given my available evidence). I do not know well other examples beyond case of Automatic Speech Recognition, but since this case caused a lot of noise, I bet it is responsible for a reasonable chunk of the Deep learning "buzz". Here is my take about this. If you look at papers from Microsoft like Seide et al 2011 and similar papers the reported improvement against state of the art (up to 30%) is really impressive and seems solid. Now, the technique is more or less using a very big multi-layer perceptron (MLP), a technique already established two decades ago (or more). There is some fancy stuff like the deep belief network based initialization, but it does not make big differences. The core of the recipe itself is not very new. What has changed is the scale of data we have available and the size of the models that we can handle. With this I am not implying that this is not a very interesting discovery. But it is important to bear in mind that the change in the amount of data could also make other 20 year old techniques interesting again. On the other hand, neural networks had a bad name in the last years for understandable reasons. They are a blackbox, or at least less transparent than the statistical methods. This makes them prone to cause the "black box delusion" effect. You hear a new algorithm is in town, it has fancy stuff like remotely resembling human thinking architectures or cool math but you can not completely grasp it guts, then "voila!" suddenly you are overestimating its relevance and scope of applicably. MLPs were hailed as "the" tool for machine learning already once, I think for these same reasons. For me the right position here is a prudent skepticism. On the other hand, this should also push people to try new/old radical stuff since the rules of the game seem to be changing, it is not a moment to be conservative in ML research :).

lrei 14y ago
I've heard this argument ever since Norvig's Unreasonable Effectiveness of Data. While having a ton of data available is great, it has its limits. I believe you are overestimating the effectiveness of data (as, imo, Norvig did). And here specifically, it's not the case for the hype: from the NYT article [1]: "The achievement was particularly impressive because the team decided to enter the contest at the last minute and designed its software with no specific knowledge about how the molecules bind to their targets. The students were also working with a relatively small set of data; neural nets typically perform well only with very large ones." NNs in general have enjoyed lots of successful practical (commercial) applications in pattern recognition though they were sort of replaced in the "state-of-the-art" by SVMs in many cases until RBMs and DBNs came along. I agree with your caution for skepticism though, only time will tell how good DBNs are. I think the black box criticism is BS for the most part. In some cases (google's search being a famous example) it might be great to have a human readable and tweakable solution (assuming you have the resources) but for something like recognising handwritten digits from images, not so much. [1] http://www.nytimes.com/2012/11/24/science/scientists-see-advances-in-deep-learning-a-part-of-artificial-intelligence.html?_r=0 http://www.nytimes.com/2012/11/24/science/scientists-see-adv...

zby 14y ago

I registered and listened for maybe 9 minutes - and nothing interesting, just talk about the talk. How I hate webcasts!

jph00 14y ago
I believe the talk about the talk only covers 3 min 20 s.
anigbrowl 14y ago
Agreed. Don't care much for podcasts either, although I can see the value for people who drive frequently.

rm999 14y ago

I'm concerned deep networks are being overhyped. They're certainly exciting, but they haven't seen much use in industry yet; it's too early to make claims about how they have impacted data science. Also, data science involves a lot more than building predictive models. In my experience >95% of effort goes into something other than building a model. In kaggle contests you usually concentrate on that <5%, which IMO is the fun part but it's not the reality of industry. There are many big breakthroughs in data science that don't involve model building. edit: I haven't listened to the podcast yet (at work), my comment is more about the title.

lrei 14y ago
I'm genuinely curious (not being snarky or wtv): what do you put 95% of your effort into?
- mshron 14y ago
  Not the OP but: * Problem definition * Infrastructure * Data transformation * Exploratory analysis (arguably part of model work) * Results presentation Then again, this is an ongoing disagreement I have with the Kaggle folks over what constitutes "data science," where I'm pretty confident that "applied machine learning" is a better explanation of what their contests are about.
  - rm999 14y ago
    I kind of got them to say it here: https://news.ycombinator.com/item?id=4655927 https://news.ycombinator.com/item?id=4655927 BTW, I'm a big fan of the data analysis that came out of okcupid, is that all your work?
  - lrei 14y ago
    I see. Thanks. I'd say data transformation is a part of feature engineering (commonly the bulk of the effort in a ML application). And exploratory analysis is part of model work. W/o those 2 one would be building a model out of dreams and wishes. Data Science is probably a poorly chosen description. I'd say common use includes infrastructure work which for most of us consists in engineering work.

Aloisius 14y ago

I would love to watch this, but O'Reilly's presentation streamer is awful. I tried jumping ahead, but the video stream doesn't actually jump with me so I end up listening to one part and watching another (tried under FireFox, Safari and Chrome on Mac). I don't suppose someone has an alternative version somewhere?

softbuilder 14y ago

This is a new topic to me. Are there any whitepapers or open source projects touching on this?

pbharrin 14y ago

You need Flash to watch this even on Chrome.

hsshah 14y ago

We should discourage submissions like this here on HN that require registration to view the content. So even though this topic is of great interest of mine, I will not be upvoting it. Sorry.

drucken 14y ago
While I agree with you, note there is already flagrant "abuse" of this on HN by the posting of newspaper links with paywalls, in particular the New York Times.

stiff 14y ago

I am sorry, but does anyone else have the impression that a lot of people is commenting here with so much confidence while clearly not knowing anything about the topic? It takes almost an equivalent of an undergraduate mathematics degree and than a lot of experience in ML itself to get a decent understanding how things like Deep Belief Networks work, so I don't wonder none of the comments so far hinted at any understanding of anything particular about deep learning, just general derogatory comments "not used in industry", "overhyped", pointers to whatever someone heard in an undergraduate ML class on older types of networks etc. Maybe if you don't have anything on topic to say, just do not comment? You really are not obliged to have an opinion on everything. (Waiting for the downvotes)

ergodic 14y ago
rather than being so aggressive and waiting for downvotes you could choose to be constructive ;) and answer a concise technical question to a sub-thread that you yourself started https://news.ycombinator.com/item?id=5377101 https://news.ycombinator.com/item?id=5377101
spikels 14y ago
It is purely a matter of opinion whether DBNs are "overhyped" but I hope you would agree that they are currently being "hyped". And I hope you understand how this can actually damage the potential of what is likely some very good technology like say like what happened to neural nets. :) It is also a matter of opinion how widely they are being used in industry. Certainly they are being studied in many companies but they do not a appear to used much in production because of their complexity and high training cost. This is still cutting edge technology. In my experience most professionally trained mathematicians and statisticians are still pretty skeptical of these claims. Wouldn't you agree?
- stiff 14y ago
  It is OK to be skeptical, all I am trying to say is that most of the comments leave the impression of the poster trying to lean over backwards to say something at all related to the topic, often ending up with generic truisms. For example how is model building being a small part of practical ML criticism of deep learning? How is it on-topic at all?

espeed 14y ago

Quoc V. Le (http://ai.stanford.edu/~quocle/ http://ai.stanford.edu/~quocle/) of Stanford and Google did a talk today at Univ. of Washington: "Scaling deep learning to 10,000 cores and beyond": Presentation Univ. of Washington (March 14, 2013) https://www.cs.washington.edu/htbin-post/mvis/mvis?ID=1338 You can see one of Quoc's previous talks online: "Tera-scale deep learning: - Quoc V. Le from ML Lunch @ CMU http://vimeo.com/52332329 http://vimeo.com/52332329 You may remember Jeff Dean's (http://research.google.com/pubs/jeff.html http://research.google.com/pubs/jeff.html) post on this: https://plus.google.com/118227548810368513262/posts/PozFb134egM https://plus.google.com/118227548810368513262/posts/PozFb134... The corresponding research at Google... "Building high-level features using large scale unsupervised learning" http://research.google.com/pubs/pub38115.html http://research.google.com/pubs/pub38115.html http://research.google.com/archive/unsupervised_icml2012.html http://research.google.com/archive/unsupervised_icml2012.htm... Previous HN discussion: https://news.ycombinator.com/item?id=4145558 https://news.ycombinator.com/item?id=4145558 -- How Many Computers to Identify a Cat? 16,000 http://www.nytimes.com/2012/06/26/technology/in-a-big-network-of-computers-evidence-of-machine-learning.html?_r=2&smid=go-share&#38 http://www.nytimes.com/2012/06/26/technology/in-a-big-networ...;

pilooch 14y ago

Deep learning is so attractive to any AI and machine learning practitioner! The results are beautiful to witness or read about. This is clearly another step in a direction that many of us have been waiting (or working on) for a long time! That said, AI, like every other sciences, experiences trends and bubbles. If you give a decent look to usages and problem solving with machine learning, deep learning techniques are not exactly the final answer. Typically, they're slow to train, to my knowledge there is no good 'online' algorithm yet to train them (i.e. for autoencoders, recursive autoencoders, Boltzman machines). Many applications, and a trend toward 'lifelong learning'[1], require fast incremental learning that yields results in near real-time, or at least in minutes rather than days. I've compared a couple of unsupervised machine learning algorithms with recursive autoencoders: the latter can learn deeper, very often, but at a computational costs (days vs seconds). Deep learning computation will improve, for sure though. [1] http://cs.brynmawr.edu/~eeaton/AAAI-SSS13-LML/ http://cs.brynmawr.edu/~eeaton/AAAI-SSS13-LML/

wfunction 14y ago

Why are they are they tricking us into giving away personal information?

amit_m 14y ago

Argh! Worst streaming experience ever. Also, no contents in the first 20 minutes. It took me a while to understand that the slides were in a popup was blocked. After reloading, the slides don't match the audio.