3 ms·

Deep learning has pushed up the state of the art exponentially in the last few years. On some benchmarks like speech recognition, deep learning methods increase

Deep learning has pushed up the state of the art exponentially in the last few years. On some benchmarks like speech recognition, deep learning methods increased the state of the art by what would have taken decades at the previous rate of progress. This years Imagenet winner achieved 5% error. Last year was 11%, the year before 15%. (Note percentage error isn't linear, each percentage point is harder than the last.) Everything from Machine vision to natural language processing to speech recognition, is benefiting from this. We live in exciting times for AI, and everyone wants to get in on it. Simple machine learning algorithms are interesting and even superior on simpler problems. But they don't scale up to complicated AI tasks like this. They are just doing simple correlations between variables, whereas NN's can theoretically learn any arbitrary function (theoretically Turing complete.) Usually this does make them overfit, but on more complicated problems underfitting is the bigger problem.

hyperbovine 12y ago
To be honest, I have never understood all the hoopla concerning the universal approximation theorem. Least squares in your favorite orthormal basis for L2[0,1] will do the same thing and it can be easier to match the correct basis with your problem domain. If there was some result showing that NNs converge at a faster rate (as a function of the number of parameters being fitted) that would be nice, but I am not aware of any. Also, universal approximation has nothing to do with Turing completeness.
- abhgh 12y ago
  With the right basis functions, yes, you would end up approximating the target function well. The problem is how do you determine the right basis functions? In SVMs, for example, there is no one good way to pick the right kernel today. This is a benefit with ANNs - the universal approximation theorem tells you that there is nothing else you need to decide on once you're using ANNs. Sure there are parameters like the no of layers etc, but those parameters are something you need account for in other techniques too. Not saying ANNs give the best solution always. Or even the best first model to try out on all problems; only pointing out that there is a convincing case to be made for them, and the universal approximation theorem has a lot to do with it. Wrt your other point, Turing completeness was proved separately - look for the paper "Turing computability with neural nets" by Siegelmann and Sontag. Not sure your parent comment intended the association.
fchollet 12y ago
> Everything from Machine vision to natural language processing to speech recognition, is benefiting from this. We live in exciting times for AI, and everyone wants to get in on it. Not "everything" is benefiting from multilayer CNNs; in fact their use is limited to a tiny fraction of machine learning problems. The majority of data problems in the industry involve small datasets with a limited number of dimensions; picture classification and speech recognition are outliers in their scale, although they are extremely important problems. Also, the fact is that multilayer CNNs are not a recent development, even though the catchy term "deep learning" is. The recent development that has brought us significant performance improvements in image classification and speech recognition is mainly to put CNNs on GPUs, allowing us to scale the networks considerably. So you could say that what is going on here is really hardware progress, not any kind of theoretical breakthrough (there is surprisingly little theory behind multilayer CNNs). At last, improvements over the state of the art on a number of large-scale ML problems is not the reason there is so much hype about "deep learning" (which is really the same kind of shallow learning that we've been doing for a while), both in the mainstream media and in the tech community. We have entered a new AI summer, where algorithms that have been around for decades are being hailed as being the "real deal" that works just like the brain (nope). A company like Vicarious has raised 50M not long ago on a promise to create "human-level vision" by 2015 and "strong AI" by 2018. Somehow I don't think investors will see much of a return on that money. This is all nice and good, but after summer comes winter. The hotter the summer the darker the winter. It has happened before, and it has hurt AI progress very much.
- cracker_jacks 12y ago
  A couple of issues with your post. >The majority of data problems in the industry involve small datasets with a limited number of dimensions This really depends on your industry. And you're forgetting much of the deep learning improvements when applied to natural language processing. >A company like Vicarious has raised 50M not long ago on a promise to create "human-level vision" by 2015 and "strong AI" by 2018. Somehow I don't think investors will see much of a return on that money. What evidence do you have that Vicarious is even using deep learning?

hyperbovine 12y ago

To be honest, I have never understood all the hoopla concerning the universal approximation theorem. Least squares in your favorite orthormal basis for L2[0,1] will do the same thing and it can be easier to match the correct basis with your problem domain. If there was some result showing that NNs converge at a faster rate (as a function of the number of parameters being fitted) that would be nice, but I am not aware of any. Also, universal approximation has nothing to do with Turing completeness.

abhgh 12y ago
With the right basis functions, yes, you would end up approximating the target function well. The problem is how do you determine the right basis functions? In SVMs, for example, there is no one good way to pick the right kernel today. This is a benefit with ANNs - the universal approximation theorem tells you that there is nothing else you need to decide on once you're using ANNs. Sure there are parameters like the no of layers etc, but those parameters are something you need account for in other techniques too. Not saying ANNs give the best solution always. Or even the best first model to try out on all problems; only pointing out that there is a convincing case to be made for them, and the universal approximation theorem has a lot to do with it. Wrt your other point, Turing completeness was proved separately - look for the paper "Turing computability with neural nets" by Siegelmann and Sontag. Not sure your parent comment intended the association.

fchollet 12y ago

> Everything from Machine vision to natural language processing to speech recognition, is benefiting from this. We live in exciting times for AI, and everyone wants to get in on it. Not "everything" is benefiting from multilayer CNNs; in fact their use is limited to a tiny fraction of machine learning problems. The majority of data problems in the industry involve small datasets with a limited number of dimensions; picture classification and speech recognition are outliers in their scale, although they are extremely important problems. Also, the fact is that multilayer CNNs are not a recent development, even though the catchy term "deep learning" is. The recent development that has brought us significant performance improvements in image classification and speech recognition is mainly to put CNNs on GPUs, allowing us to scale the networks considerably. So you could say that what is going on here is really hardware progress, not any kind of theoretical breakthrough (there is surprisingly little theory behind multilayer CNNs). At last, improvements over the state of the art on a number of large-scale ML problems is not the reason there is so much hype about "deep learning" (which is really the same kind of shallow learning that we've been doing for a while), both in the mainstream media and in the tech community. We have entered a new AI summer, where algorithms that have been around for decades are being hailed as being the "real deal" that works just like the brain (nope). A company like Vicarious has raised 50M not long ago on a promise to create "human-level vision" by 2015 and "strong AI" by 2018. Somehow I don't think investors will see much of a return on that money. This is all nice and good, but after summer comes winter. The hotter the summer the darker the winter. It has happened before, and it has hurt AI progress very much.

cracker_jacks 12y ago
A couple of issues with your post. >The majority of data problems in the industry involve small datasets with a limited number of dimensions This really depends on your industry. And you're forgetting much of the deep learning improvements when applied to natural language processing. >A company like Vicarious has raised 50M not long ago on a promise to create "human-level vision" by 2015 and "strong AI" by 2018. Somehow I don't think investors will see much of a return on that money. What evidence do you have that Vicarious is even using deep learning?