4 ms·

I think we are at a tipping point for machine learning. Deep learning is just getting ramped up and the results are extremely impressive. [1] How far this coul

by dia80 14y ago

I think we are at a tipping point for machine learning. Deep learning is just getting ramped up and the results are extremely impressive. [1] How far this could go is hard to say right now but things like this [2] (10 million images, 1 week training, 16,000 cores) are bringing enormous processing power to bear for the first time on these kind of problems. Is anyone able to comment on what moving to custom hardware will mean for performance on these kind of tasks? [1] http://www.cs.toronto.edu/~hinton/absps/imagenet.pdf http://www.cs.toronto.edu/~hinton/absps/imagenet.pdf [2] http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/archive/large_deep_networks_nips2012.pdf http://static.googleusercontent.com/external_content/untrust...

gcr 14y ago
It's my understanding that deep learning networks have to be tuned almost perfectly to perform well. Not my area ef expertise, but I'm not convinced just yet.
- dia80 14y ago
  In fact almost the opposite! Unsupervised pre-training learns the statistical invariants in the inputs and these become the features. This is in contrast to hand engineering features for your particular domain which, unsurprisingly, doesn't generalise well.
- SatvikBeri 14y ago
  It's true that Deep Belief Networks take a lot of work compared to, say, Random Forests, but they make good Neural Networks much easier. Before Deep Learning, Neural Networks required a lot of hand tuning. Without that kind of tuning, Neural Networks with more than 1 or 2 hidden layers tended to perform very badly, because they would get stuck in plateaus or poor local optima. In contrast, (one facet of) Deep Learning focuses on using unsupervised learning to better initialize NNs, which in turn allows for much higher levels of accuracy. In essence, Deep Learning helps pick the NN parameters automatically. So it's actually decreasing the amount of hand tuning needed.
- dumitrue 14y ago
  Basically not true (anymore). If you take the model that is described by Hinton et al and apply it to some vision task for which you generated a lot of data, it will invariably work better than the competition, almost out of the box (yeah you have to try a few learning rates, but that basically the case for any ML algorithm). It's pretty amazing actually.
dumitrue 14y ago
So actually if you read Geoff Hinton's paper on how they won the ImageNet challenge, they use 2 GPUs. Whether this qualifies as "custom hardware" or not, it's debatable, but it's certainly striking that you can obtain performance that is about as good as the 16k cores result. The results and the models are not directly comparable, but still. One limitation of using GPUs is the fact that they have comparatively little on-board memory (max 6GB for now) and shuffling data in/out of it is expensive (because of communication costs). Now there's also the idea that you can maybe obtain something interesting from a learning perspective when training on 16k cores. Because you have to essentially train a bunch of different models, that have different parameters and see potentially different subsets of your training set, ultimately you perform some sort of model averaging (since training is doing it for you). In essence, by doing distributed gradient descent you are potentially training a clever ensemble of networks, which arguably could give better generalization. Can you have an ensemble of GPUs? Now that's an interesting question too :)
- dia80 14y ago
  I was thinking about ASIC? Here is a bitcoin mining solution: http://www.butterflylabs.com/ http://www.butterflylabs.com/
  - dumitrue 14y ago
    Check this out, if you're curious about this line of thought: http://yann.lecun.com/exdb/publis/pdf/farabet-fpl-09.pdf http://yann.lecun.com/exdb/publis/pdf/farabet-fpl-09.pdf The ImageNet winning model is a essentially glorified ConvNet :)
rm999 14y ago
I'd argue we hit the tipping point more than ten years ago. Finance and a lot of tech (e.g. google) had already been transformed by machine learning at that point. At my last job we were working with some pretty complex neural network models that were first developed in the early to mid 90s. AI research died off in the 1960s and 1970s because the results didn't even get close to matching the expectations. Machine learning's current commercial success ensures that won't happen in the foreseeable future. That said, it's worth tempering our enthusiasm. To put it perspective, deep networks are extremely far from recreating general, human-level intelligence. There's a long way to go, and I'm not convinced the path we're on is the 'correct' one to reaching the kind of general AI people see in the movies.
- eli_gottlieb 14y ago
  I was under the impression that "Turing Test-passing general sapience" was actually a pretty niche field in AI/ML these days. Most everyone seems to have found it cheaper to build a special-purpose machine learning engine/algorithm for their particular problem than to wait for a speaking, reasoning human-equivalent computer.
  - Houshalter 14y ago
    True general artificial intelligence would be capable of actually solving problems. Which means it could write code, engineer things, maybe even improve itself, etc. Most machine learning is limited to mindlessly exploring a search space which makes it good mostly at pattern recognition. But they are pretty terrible at solving difficult problems or writing code and things like that. Including special purpose algorithms. We have the learning part down (somewhat), but the intelligence part not so much.

by dia80 14y ago

gcr 14y ago

It's my understanding that deep learning networks have to be tuned almost perfectly to perform well. Not my area ef expertise, but I'm not convinced just yet.

dia80 14y ago
In fact almost the opposite! Unsupervised pre-training learns the statistical invariants in the inputs and these become the features. This is in contrast to hand engineering features for your particular domain which, unsurprisingly, doesn't generalise well.
SatvikBeri 14y ago
It's true that Deep Belief Networks take a lot of work compared to, say, Random Forests, but they make good Neural Networks much easier. Before Deep Learning, Neural Networks required a lot of hand tuning. Without that kind of tuning, Neural Networks with more than 1 or 2 hidden layers tended to perform very badly, because they would get stuck in plateaus or poor local optima. In contrast, (one facet of) Deep Learning focuses on using unsupervised learning to better initialize NNs, which in turn allows for much higher levels of accuracy. In essence, Deep Learning helps pick the NN parameters automatically. So it's actually decreasing the amount of hand tuning needed.
dumitrue 14y ago
Basically not true (anymore). If you take the model that is described by Hinton et al and apply it to some vision task for which you generated a lot of data, it will invariably work better than the competition, almost out of the box (yeah you have to try a few learning rates, but that basically the case for any ML algorithm). It's pretty amazing actually.

dumitrue 14y ago

So actually if you read Geoff Hinton's paper on how they won the ImageNet challenge, they use 2 GPUs. Whether this qualifies as "custom hardware" or not, it's debatable, but it's certainly striking that you can obtain performance that is about as good as the 16k cores result. The results and the models are not directly comparable, but still. One limitation of using GPUs is the fact that they have comparatively little on-board memory (max 6GB for now) and shuffling data in/out of it is expensive (because of communication costs). Now there's also the idea that you can maybe obtain something interesting from a learning perspective when training on 16k cores. Because you have to essentially train a bunch of different models, that have different parameters and see potentially different subsets of your training set, ultimately you perform some sort of model averaging (since training is doing it for you). In essence, by doing distributed gradient descent you are potentially training a clever ensemble of networks, which arguably could give better generalization. Can you have an ensemble of GPUs? Now that's an interesting question too :)

dia80 14y ago
I was thinking about ASIC? Here is a bitcoin mining solution: http://www.butterflylabs.com/ http://www.butterflylabs.com/
- dumitrue 14y ago
  Check this out, if you're curious about this line of thought: http://yann.lecun.com/exdb/publis/pdf/farabet-fpl-09.pdf http://yann.lecun.com/exdb/publis/pdf/farabet-fpl-09.pdf The ImageNet winning model is a essentially glorified ConvNet :)

rm999 14y ago

I'd argue we hit the tipping point more than ten years ago. Finance and a lot of tech (e.g. google) had already been transformed by machine learning at that point. At my last job we were working with some pretty complex neural network models that were first developed in the early to mid 90s. AI research died off in the 1960s and 1970s because the results didn't even get close to matching the expectations. Machine learning's current commercial success ensures that won't happen in the foreseeable future. That said, it's worth tempering our enthusiasm. To put it perspective, deep networks are extremely far from recreating general, human-level intelligence. There's a long way to go, and I'm not convinced the path we're on is the 'correct' one to reaching the kind of general AI people see in the movies.

eli_gottlieb 14y ago
I was under the impression that "Turing Test-passing general sapience" was actually a pretty niche field in AI/ML these days. Most everyone seems to have found it cheaper to build a special-purpose machine learning engine/algorithm for their particular problem than to wait for a speaking, reasoning human-equivalent computer.
- Houshalter 14y ago
  True general artificial intelligence would be capable of actually solving problems. Which means it could write code, engineer things, maybe even improve itself, etc. Most machine learning is limited to mindlessly exploring a search space which makes it good mostly at pattern recognition. But they are pretty terrible at solving difficult problems or writing code and things like that. Including special purpose algorithms. We have the learning part down (somewhat), but the intelligence part not so much.