5 ms·

Large Scale Distributed Deep Networks (by Jeff Dean et al)

michael_nielsen 14y ago
Also submitted yesterday, by someone else, but it got buried: http://news.ycombinator.com/item?id=4775644 http://news.ycombinator.com/item?id=4775644 I resubmitted since I'm pretty sure this is of interest to many HN readers. Examples of why it's interesting include: 1. Google's deep learning work is now being used to power Android voice search (http://googleresearch.blogspot.ca/2012/08/speech-recognition-and-deep-learning.html http://googleresearch.blogspot.ca/2012/08/speech-recognition... ); and 2. Dean claims that "We are seeing better than human-level performance in some visual tasks," in particular, for the problem of extracting house numbers in photos taken by Google's Street View cars, a job that used to be done by a large team of people (http://www.technologyreview.com/news/429442/google-puts-its-virtual-brain-technology-to-work/ http://www.technologyreview.com/news/429442/google-puts-its-... ).
- dumitrue 14y ago
  It's not obvious to me what it means to have "better than human-level performance" since most of the time the ground-truth itself is defined by humans :)
  - dsl 14y ago
    One example I can think of is a computer could read a house number or a street sign from 100 feet away, where a human with good vision might be able to make out the same text at 20 feet.
  - jasonwatkinspdx 14y ago
    In production you may use a process that's more economized than what you'd use to establish a ground truth in research. So the human team in production will have some error rate that is tolerated or won't include as much redundancy. In this sense it's easy to imagine better than human performance from the software, in the sense that the human performance you get isn't a single max value, but rather a function of budget.
  - robrenaud 14y ago
    You could also ask many humans and average their response to obtain a golden label, and see how well any particular human agrees with the average. If there is a lot of variance in the human answers, then it's possible for a machine to have better than (individual) human performance, even on a human labelled data set.
  - fudged71 14y ago
    CATCHAS, ironically, often seem to require "better than human-level performance" for recognition
- ArbitraryLimits 14y ago
  > "We are seeing better than human-level performance in some visual tasks," in particular, for the problem of extracting house numbers in photos taken by Google's Street View cars Can someone explain to me how this is news, given that the handwritten addresses on snail mail envelopes in the US have been OCR'd by neural networks for more than twenty years now?
  - confluence 14y ago
    The order of complexity difference between recognising figures on plain paper perpendicular to a scanning device in a controlled environment, and doing the same thing on huge amounts of non-standard chaotic data is why.
    - ArbitraryLimits 14y ago
      Virtually all house numbers are either painted from a stencil or composed of mass-produced shapes on a background of uniform color, whereas addresses on envelopes are handwritten by doctors, six-year-olds and people with Parkinson's disease. I'm not convinced it's a harder problem.
      - confluence 14y ago
        What don't you get? One is on a white background. One is in random orientations, placed in complex scenes, with random fonts, positions, numbers, sizes, shapes and locations and you don't even know where they are. It's like a game of "Where's Waldo" on freaking crack. You have literally no idea how complex this stuff is now then do you?
        ArbitraryLimits 14y ago
        Compare the addresses at http://www.realsimple.com/home-organizing/decorating/eye-catching-house-numbers-10000001218771/index.html http://www.realsimple.com/home-organizing/decorating/eye-cat... and http://mandydouglass.blogspot.com/2010/10/addressing-envelopes.html http://mandydouglass.blogspot.com/2010/10/addressing-envelop... . Those are two representative images I picked from the first google hits for "house number" and "handwritten address" respectively - all the others were comparable. Are you seriously going to claim that the house number is harder to recognize than the handwriting? I do have an idea (literally, even) that there are additional problems having to do with extracting the house number images themselves from full-motion video, but that's an image registration problem and not an object recognition problem.
bravura 14y ago
I did my postdoctoral research on deep learning, and got into it back when Geoff Hinton's work was just an unpublished tech report. So if anyone has any questions on it, I will try to answer.
- e98cuenc 14y ago
  Is there any open source ML framework that includes support for DNN? Do you know of any tutorial that may guide the beginner using DNN, I have no idea how to choose the number of hidden layers and activation functions. Thanks!
  - dumitrue 14y ago
    http://deeplearning.net/software/theano/ http://deeplearning.net/software/theano/ is a good place to start. It's open source, in Python, has a few tutorials that lead you towards some rather state-of-the-art methods. There's no secret sauce for how to choose the number of layers and the activation functions (and anyone that tells you otherwise is lying). It's all application-dependent and the best way to do it is by cross-validation. Can answer questions about this topic in conjunction with mr. bravura (I'm working with the code referenced in that paper by Dean et al., here at Google, and did grad school at the same place as bravura did his postdoc).
    - e98cuenc 14y ago
      I'm a xoogler, stuff like this makes me wish I was back there :) I want to play a bit with this stuff and have some fun, so thanks for the Theano reference, seems cool! WRT the DNN parameters, is it possible to try all potential possibilities (within reason) and find the best one using only cross-validation, or are there just too many choices and you have to use intuition? (from your comment I don't get if cross-validation is ok to get optimal number of layers etc. or if you have to be "smart") Thanks for replying!
      - dumitrue 14y ago
        For things like the number of layers, the total number of options is relatively small -- usually people try between 1 and 6-7. For most of the other parameters you have to be smarter than that, especially since a lot of them are real-valued, so you can't really explore them all. One of the trends these days is to perform automatic hyper-parameter tuning, especially for cases where a full exploration of hyper-parameters via grid-search would mean a combinatorial explosion of possibilities (and for neural networks you can conceivably explore dozen of hyper-parameters). A friend of mine just got a paper published at NIPS (same conference) on using Bayesian optimization/Gaussian processes for optimizing the hyper-parameters of a model -- http://www.dmi.usherb.ca/~larocheh/publications/gpopt_nips.pdf http://www.dmi.usherb.ca/~larocheh/publications/gpopt_nips.p.... They get better than state of the art results on a couple of benchmarks, which is neat. The code is public -- http://www.cs.toronto.edu/~jasper/software.html http://www.cs.toronto.edu/~jasper/software.html -- and in python, so you could potentially try it out (runs on EC2, too). Btw, Geoff Hinton is teaching an introductory neural nets class on Coursera these days, you should check it out, he's a great teacher. Also, you can always come back to Google, we're doing cool stuff with this :)
feniv 14y ago
This is written by THE Jeff Dean, who is essentially the programming equivalent of Chuck Norris at Google ( http://www.quora.com/Jeff-Dean/What-are-all-the-Jeff-Dean-facts http://www.quora.com/Jeff-Dean/What-are-all-the-Jeff-Dean-fa... )
samg_ 14y ago
A year ago, I would have no idea what this was about. I'm very thankful that I live in a world where I can take high quality classes from Coursera that have given me the foundation to at least understand the abstract. :)
- junto 14y ago
  Which courses did you take out of interest? I struggle with the mathematics used in neural networks. I can understand code but as soon as I start to see calculus my brain freezes over. Does anyone know of a good online course that can give me a crash course in the mathematics required for neural networks? My CS bachelors covered this, but I was lazy and drank too much beer. Now 20 years later I want to understand it properly.
  - dave_sullivan 14y ago
    Do the theano deep learning tutorials. And keep hacking away at the math--you need it, but it eventually sinks in and becomes reasonably intuitive. Starting with code helped me grasp the math (I'm also much more comfortable reading code than math).
  - samg_ 14y ago
    I've taken Andrew Ng's Machine Learning class, Daphne Koller's Probabilistic Graphical Models class, Dan Jurafsky and Christopher Manning's Natural Language Processing class, and currently Geoff Hinton's Neural Networks class. I have spent a lot of time on Khan Academy to learn the calculus. In my experience you can get by with a surprisingly small amount of calculus, but it happens to be a small amount from a high level. For example, backpropagation is just repeated application of the chain rule. Did take a while to get a handle on the derivatives, but it's worth it.
deleted 14y ago
[deleted]

Also submitted yesterday, by someone else, but it got buried: http://news.ycombinator.com/item?id=4775644 http://news.ycombinator.com/item?id=4775644 I resubmitted since I'm pretty sure this is of interest to many HN readers. Examples of why it's interesting include: 1. Google's deep learning work is now being used to power Android voice search (http://googleresearch.blogspot.ca/2012/08/speech-recognition-and-deep-learning.html http://googleresearch.blogspot.ca/2012/08/speech-recognition... ); and 2. Dean claims that "We are seeing better than human-level performance in some visual tasks," in particular, for the problem of extracting house numbers in photos taken by Google's Street View cars, a job that used to be done by a large team of people (http://www.technologyreview.com/news/429442/google-puts-its-virtual-brain-technology-to-work/ http://www.technologyreview.com/news/429442/google-puts-its-... ).

dumitrue 14y ago
It's not obvious to me what it means to have "better than human-level performance" since most of the time the ground-truth itself is defined by humans :)
- dsl 14y ago
  One example I can think of is a computer could read a house number or a street sign from 100 feet away, where a human with good vision might be able to make out the same text at 20 feet.
- jasonwatkinspdx 14y ago
  In production you may use a process that's more economized than what you'd use to establish a ground truth in research. So the human team in production will have some error rate that is tolerated or won't include as much redundancy. In this sense it's easy to imagine better than human performance from the software, in the sense that the human performance you get isn't a single max value, but rather a function of budget.
- robrenaud 14y ago
  You could also ask many humans and average their response to obtain a golden label, and see how well any particular human agrees with the average. If there is a lot of variance in the human answers, then it's possible for a machine to have better than (individual) human performance, even on a human labelled data set.
- fudged71 14y ago
  CATCHAS, ironically, often seem to require "better than human-level performance" for recognition
ArbitraryLimits 14y ago
> "We are seeing better than human-level performance in some visual tasks," in particular, for the problem of extracting house numbers in photos taken by Google's Street View cars Can someone explain to me how this is news, given that the handwritten addresses on snail mail envelopes in the US have been OCR'd by neural networks for more than twenty years now?
- confluence 14y ago
  The order of complexity difference between recognising figures on plain paper perpendicular to a scanning device in a controlled environment, and doing the same thing on huge amounts of non-standard chaotic data is why.
  - ArbitraryLimits 14y ago
    Virtually all house numbers are either painted from a stencil or composed of mass-produced shapes on a background of uniform color, whereas addresses on envelopes are handwritten by doctors, six-year-olds and people with Parkinson's disease. I'm not convinced it's a harder problem.
    - confluence 14y ago
      What don't you get? One is on a white background. One is in random orientations, placed in complex scenes, with random fonts, positions, numbers, sizes, shapes and locations and you don't even know where they are. It's like a game of "Where's Waldo" on freaking crack. You have literally no idea how complex this stuff is now then do you?
      - ArbitraryLimits 14y ago
        Compare the addresses at http://www.realsimple.com/home-organizing/decorating/eye-catching-house-numbers-10000001218771/index.html http://www.realsimple.com/home-organizing/decorating/eye-cat... and http://mandydouglass.blogspot.com/2010/10/addressing-envelopes.html http://mandydouglass.blogspot.com/2010/10/addressing-envelop... . Those are two representative images I picked from the first google hits for "house number" and "handwritten address" respectively - all the others were comparable. Are you seriously going to claim that the house number is harder to recognize than the handwriting? I do have an idea (literally, even) that there are additional problems having to do with extracting the house number images themselves from full-motion video, but that's an image registration problem and not an object recognition problem.

bravura 14y ago

I did my postdoctoral research on deep learning, and got into it back when Geoff Hinton's work was just an unpublished tech report. So if anyone has any questions on it, I will try to answer.

e98cuenc 14y ago
Is there any open source ML framework that includes support for DNN? Do you know of any tutorial that may guide the beginner using DNN, I have no idea how to choose the number of hidden layers and activation functions. Thanks!
- dumitrue 14y ago
  http://deeplearning.net/software/theano/ http://deeplearning.net/software/theano/ is a good place to start. It's open source, in Python, has a few tutorials that lead you towards some rather state-of-the-art methods. There's no secret sauce for how to choose the number of layers and the activation functions (and anyone that tells you otherwise is lying). It's all application-dependent and the best way to do it is by cross-validation. Can answer questions about this topic in conjunction with mr. bravura (I'm working with the code referenced in that paper by Dean et al., here at Google, and did grad school at the same place as bravura did his postdoc).
  - e98cuenc 14y ago
    I'm a xoogler, stuff like this makes me wish I was back there :) I want to play a bit with this stuff and have some fun, so thanks for the Theano reference, seems cool! WRT the DNN parameters, is it possible to try all potential possibilities (within reason) and find the best one using only cross-validation, or are there just too many choices and you have to use intuition? (from your comment I don't get if cross-validation is ok to get optimal number of layers etc. or if you have to be "smart") Thanks for replying!
    - dumitrue 14y ago
      For things like the number of layers, the total number of options is relatively small -- usually people try between 1 and 6-7. For most of the other parameters you have to be smarter than that, especially since a lot of them are real-valued, so you can't really explore them all. One of the trends these days is to perform automatic hyper-parameter tuning, especially for cases where a full exploration of hyper-parameters via grid-search would mean a combinatorial explosion of possibilities (and for neural networks you can conceivably explore dozen of hyper-parameters). A friend of mine just got a paper published at NIPS (same conference) on using Bayesian optimization/Gaussian processes for optimizing the hyper-parameters of a model -- http://www.dmi.usherb.ca/~larocheh/publications/gpopt_nips.pdf http://www.dmi.usherb.ca/~larocheh/publications/gpopt_nips.p.... They get better than state of the art results on a couple of benchmarks, which is neat. The code is public -- http://www.cs.toronto.edu/~jasper/software.html http://www.cs.toronto.edu/~jasper/software.html -- and in python, so you could potentially try it out (runs on EC2, too). Btw, Geoff Hinton is teaching an introductory neural nets class on Coursera these days, you should check it out, he's a great teacher. Also, you can always come back to Google, we're doing cool stuff with this :)

feniv 14y ago

This is written by THE Jeff Dean, who is essentially the programming equivalent of Chuck Norris at Google ( http://www.quora.com/Jeff-Dean/What-are-all-the-Jeff-Dean-facts http://www.quora.com/Jeff-Dean/What-are-all-the-Jeff-Dean-fa... )

samg_ 14y ago

A year ago, I would have no idea what this was about. I'm very thankful that I live in a world where I can take high quality classes from Coursera that have given me the foundation to at least understand the abstract. :)

junto 14y ago
Which courses did you take out of interest? I struggle with the mathematics used in neural networks. I can understand code but as soon as I start to see calculus my brain freezes over. Does anyone know of a good online course that can give me a crash course in the mathematics required for neural networks? My CS bachelors covered this, but I was lazy and drank too much beer. Now 20 years later I want to understand it properly.
- dave_sullivan 14y ago
  Do the theano deep learning tutorials. And keep hacking away at the math--you need it, but it eventually sinks in and becomes reasonably intuitive. Starting with code helped me grasp the math (I'm also much more comfortable reading code than math).
- samg_ 14y ago
  I've taken Andrew Ng's Machine Learning class, Daphne Koller's Probabilistic Graphical Models class, Dan Jurafsky and Christopher Manning's Natural Language Processing class, and currently Geoff Hinton's Neural Networks class. I have spent a lot of time on Khan Academy to learn the calculus. In my experience you can get by with a surprisingly small amount of calculus, but it happens to be a small amount from a high level. For example, backpropagation is just repeated application of the chain rule. Did take a while to get a handle on the derivatives, but it's worth it.

deleted 14y ago

[deleted]