7 ms·

Because it's been just about 10 years since they figured out how to: 1) Use convolutional layers, ReLUs, and a few other tricks to ameliorate the vanishing gra

Because it's been just about 10 years since they figured out how to: 1) Use convolutional layers, ReLUs, and a few other tricks to ameliorate the vanishing gradient problem, 2) Perform continuous, high-dimensional stochastic gradient descent on graphics cards, and 3) Apply these things via stochastic grad-student descent to sufficiently massive datasets that even the most brute-force models and training methods (backpropagation of errors on a loss functional) can actually work. In the meantime, the hardware for doing it has become commoditized and the software has consolidated and become standardized. So now it's A Thing in industry. There are lots of "smarter" algorithms that almost definitely come closer to human cognition, for instance probabilistic program induction. But those aren't fast, and don't always neatly separate training from prediction: you're just not gonna be able to train those models ahead-of-time on a 10,000 image corpus inside a single week with today's hardware and software. We need to find ways to make machine learning fast even when it's not just a bunch of matrix multiplies. Otherwise, every time we make our algorithms more interesting, we cripple ourselves computationally.

orthoganol 10y ago
> There are lots of "smarter" algorithms that almost definitely come closer to human cognition Source?
- eli_gottlieb 10y ago
  http://science.sciencemag.org/content/350/6266/1332 http://science.sciencemag.org/content/350/6266/1332
rspeer 10y ago
I'm concerned about something similar, where a lot of AI techniques seem to be rushing toward a local maximum: * AI researchers do things that get good results faster out of Nvidia cards * Nvidia makes their cards faster at the things AI researchers are doing It's getting good results. We sure are going up this gradient quickly. But I don't think it's going to get us to a global maximum.
- fossuser 10y ago
  Human consciousness itself is probably a local maximum. Not necessarily a bad thing to see how far a certain process can get you as long as you keep getting results. At least until there's some other insight along the way.
  - AstralStorm 10y ago
    If not countered, we have a genetic algorithm that may attempt to move our far descendants out of the local maximum. Plus we can change the environment, both locally and just move into space.
- dave_sullivan 10y ago
  I agree w/ you about moving up a gradient quickly w/ the "GPU manufacturing <-> deep learning research" feedback loop. I think it could last a while though. One really important area of research is figuring out how to take better advantage of greater capacity. Also, how to do more with fewer training samples (0 shot, 1 shot, etc learning). Then there's reducing the precision of the units you're using to increase capacity through software. Applying these algos to video, audio, media generation, and others will eat up all the resources you can throw at it; the algos today could take advantage of larger capacity when applied to time-series. There's so much going on that I don't see it slowing down for at least 3-5 years. Also, I'd like to point out that we've seen some big breakthroughs in the past 10 years. But for the past 10 years, the whole field of deep learning has been looked at with skepticism and has been very niche. Over the past couple of years, a lot of money and resources have been put in place to pursue this area of research. More money doesn't necessarily mean more results, but there are many many more people working on these problems than ever before. A lot of them are legitimately brilliant researchers in the prime of their careers. I think there's still more to see. I am concerned about an Nvidia monopoly around deep learning hardware. They give away tons of free cards to deep learning research groups, but at some point they'll want people to start buying. I assume they expect that will be the enterprise set, but if suddenly they manage to move all their capable deep learning cards to Teslas only (which have a huge markup), it will put the hobbyist deep learning developer at a big disadvantage. The only check and balance on that is the fact that Nvidia makes cards for gamers too, and gaming card competition is still somewhat robust, so any technology that gives enterprise a big boost will probably make its way to their flagship gaming cards quickly. Nvidia's only real competition, AMD, is so far behind that they might as well not be in the business. As someone who usually roots for the underdog, it pains me to see AMD fumble so badly in this whole area. Quantum computing could be a new hill, but I think that's a ways out and I don't know enough about the topic to speak with any real confidence.
asimuvPR 10y ago
4) Frameworks and libraries have improved a lot and are available as open source. Having a lower barrier of entry helps.
tnecniv 10y ago
I'm definitely going to use that stochastic grad student descent line.
- eli_gottlieb 10y ago
  It's a standard-issue joke at this point ;-).
hooloovoo_zoo 10y ago
Convolutional layers have been around longer than that and don't have much to do with the vanishing gradient problem, which was never that big a problem to begin with. ReLUs are helpful, but they do not lead to THAT big of a speedup. Convolutional layers have definitely had a big impact for accuracy in image problems and sound problems. I think #2 is very important; they speed up both training and evaluation by a factor of 100 compared to a modern CPU. None of these developments really explain why AI has become so popular though. Neural nets don't actually perform very well for most problems (image and speech problems being the major exceptions) compared to tree-based methods. I think it has more to do with the development of easy-to-use machine learning libraries as well as distributed processing libraries and hardware.
- kordless 10y ago
  Consciousness of Things is an inevitable conclusion for humanity. The singularity is upon us with the advent of man merging into machine. We are driving that growth to go to the next level. Also, containerization.
  - glial 10y ago
    Most of what we call 'AI' today is just fancy regression methods. No need to be so grandiose.
    - archgoon 10y ago
      I'm pretty sure that comment was completely facetious. Note that The Singularity got lumped together with Docker containers.
      - kordless 10y ago
        Don't forget CoT vis-à-vis IoT.
  - R_haterade 10y ago
    I lol'ed. Thank you for this.
R_haterade 10y ago
I've been thinking a lot about this lately, especially as I've been learning more about the difference between "nice" optimization problems with differentiable cost functions, and "messy" optimizations with less ideal cost functions. Is there any reading material you can recommend on the recent history of ML/AI for someone like me, with a dilettante's understanding of these problems and an interest in optimization?
- jabl 10y ago
  https://arxiv.org/abs/1404.7828 https://arxiv.org/abs/1404.7828 (Disclaimer: I don't know that much about ML/Deep Learning, the above was recommended by deeplearning.net and is what I'm slogging through ATM. I do have some background in "classical" statistics & optimization theory, and the above article is not that math-heavy)
  - R_haterade 10y ago
    Thank you!
ertand 10y ago
Quantum computing can technically create a complete paradigm shift as it increases to computing power drastically. well, in theory at least

orthoganol 10y ago

> There are lots of "smarter" algorithms that almost definitely come closer to human cognition Source?

eli_gottlieb 10y ago
http://science.sciencemag.org/content/350/6266/1332 http://science.sciencemag.org/content/350/6266/1332

rspeer 10y ago

I'm concerned about something similar, where a lot of AI techniques seem to be rushing toward a local maximum: * AI researchers do things that get good results faster out of Nvidia cards * Nvidia makes their cards faster at the things AI researchers are doing It's getting good results. We sure are going up this gradient quickly. But I don't think it's going to get us to a global maximum.

fossuser 10y ago
Human consciousness itself is probably a local maximum. Not necessarily a bad thing to see how far a certain process can get you as long as you keep getting results. At least until there's some other insight along the way.
- AstralStorm 10y ago
  If not countered, we have a genetic algorithm that may attempt to move our far descendants out of the local maximum. Plus we can change the environment, both locally and just move into space.
dave_sullivan 10y ago
I agree w/ you about moving up a gradient quickly w/ the "GPU manufacturing <-> deep learning research" feedback loop. I think it could last a while though. One really important area of research is figuring out how to take better advantage of greater capacity. Also, how to do more with fewer training samples (0 shot, 1 shot, etc learning). Then there's reducing the precision of the units you're using to increase capacity through software. Applying these algos to video, audio, media generation, and others will eat up all the resources you can throw at it; the algos today could take advantage of larger capacity when applied to time-series. There's so much going on that I don't see it slowing down for at least 3-5 years. Also, I'd like to point out that we've seen some big breakthroughs in the past 10 years. But for the past 10 years, the whole field of deep learning has been looked at with skepticism and has been very niche. Over the past couple of years, a lot of money and resources have been put in place to pursue this area of research. More money doesn't necessarily mean more results, but there are many many more people working on these problems than ever before. A lot of them are legitimately brilliant researchers in the prime of their careers. I think there's still more to see. I am concerned about an Nvidia monopoly around deep learning hardware. They give away tons of free cards to deep learning research groups, but at some point they'll want people to start buying. I assume they expect that will be the enterprise set, but if suddenly they manage to move all their capable deep learning cards to Teslas only (which have a huge markup), it will put the hobbyist deep learning developer at a big disadvantage. The only check and balance on that is the fact that Nvidia makes cards for gamers too, and gaming card competition is still somewhat robust, so any technology that gives enterprise a big boost will probably make its way to their flagship gaming cards quickly. Nvidia's only real competition, AMD, is so far behind that they might as well not be in the business. As someone who usually roots for the underdog, it pains me to see AMD fumble so badly in this whole area. Quantum computing could be a new hill, but I think that's a ways out and I don't know enough about the topic to speak with any real confidence.

asimuvPR 10y ago

4) Frameworks and libraries have improved a lot and are available as open source. Having a lower barrier of entry helps.

tnecniv 10y ago

I'm definitely going to use that stochastic grad student descent line.

eli_gottlieb 10y ago
It's a standard-issue joke at this point ;-).

hooloovoo_zoo 10y ago

Convolutional layers have been around longer than that and don't have much to do with the vanishing gradient problem, which was never that big a problem to begin with. ReLUs are helpful, but they do not lead to THAT big of a speedup. Convolutional layers have definitely had a big impact for accuracy in image problems and sound problems. I think #2 is very important; they speed up both training and evaluation by a factor of 100 compared to a modern CPU. None of these developments really explain why AI has become so popular though. Neural nets don't actually perform very well for most problems (image and speech problems being the major exceptions) compared to tree-based methods. I think it has more to do with the development of easy-to-use machine learning libraries as well as distributed processing libraries and hardware.

kordless 10y ago
Consciousness of Things is an inevitable conclusion for humanity. The singularity is upon us with the advent of man merging into machine. We are driving that growth to go to the next level. Also, containerization.
- glial 10y ago
  Most of what we call 'AI' today is just fancy regression methods. No need to be so grandiose.
  - archgoon 10y ago
    I'm pretty sure that comment was completely facetious. Note that The Singularity got lumped together with Docker containers.
    - kordless 10y ago
      Don't forget CoT vis-à-vis IoT.
- R_haterade 10y ago
  I lol'ed. Thank you for this.

R_haterade 10y ago

I've been thinking a lot about this lately, especially as I've been learning more about the difference between "nice" optimization problems with differentiable cost functions, and "messy" optimizations with less ideal cost functions. Is there any reading material you can recommend on the recent history of ML/AI for someone like me, with a dilettante's understanding of these problems and an interest in optimization?

jabl 10y ago
https://arxiv.org/abs/1404.7828 https://arxiv.org/abs/1404.7828 (Disclaimer: I don't know that much about ML/Deep Learning, the above was recommended by deeplearning.net and is what I'm slogging through ATM. I do have some background in "classical" statistics & optimization theory, and the above article is not that math-heavy)
- R_haterade 10y ago
  Thank you!

ertand 10y ago

Quantum computing can technically create a complete paradigm shift as it increases to computing power drastically. well, in theory at least