4 ms·

FYI This post is about deep learning. It could be the case that neural networks stop getting so much hype soon, but the biggest driver of the current "AI" (ugh

by imh 8y ago

FYI This post is about deep learning. It could be the case that neural networks stop getting so much hype soon, but the biggest driver of the current "AI" (ugh I hate the term) boom is the fact that everything happens on computers now, and that isn't changing any time soon. We log everything and are even starting to automate decisions. Statistics, machine learning, and econometrics are booming fields. To talk about two topics dear to my heart, we're getting way better at modeling uncertainty (bayesianism is cool now, and resampling-esque procedures aged really well with a few decades of cheaper compute) and we're better at not only talking about what causes what (causal inference), but what causes what when (heterogeneous treatment effect estimation, e.g. giving you aspirin right now does something different from giving me aspirin now). We're learning to learn those things super efficiently (contextual bandits and active learning). The current data science boom goes far far far far beyond deep learning, and most of the field is doing great. Maybe those bits will even get better faster if deep learning stops hogging the glory. More likely, we'll learn to combine these things in cool ways (as is happening now).

digitalzombie 8y ago
Bayesian can be seen as a subset of deep learning or hell a superset. AI is a superset and Machine learning is a subset of AI and most funding is in deep learning. Once Deep Learning hit the limit I believe there will be an AI winter. Maybe there will be hype around statistic (cross fingers) which will lead to Bayesian and such.
- johnmoberg 8y ago
  How can Bayesian stuff be seen as a subset or superset of deep learning?
  - kamaal 8y ago
    I guess the point that digitalzombie is trying to make is most of what we call AI or ML or even Deep learning is simply extension of statistics on computers. Things like the German tank problem or the problem of hardening airplanes during WW2 have that very AI'esque feel to it. Where you use data to build a model, then let that data from the model to change the model as it fits. Also the whole thing about 'decision making' is either bayesian or frequency based models in nature. Most of these algorithms and math has long existed before the current boom. Its just that the raw computing power and resources that you have today make it possible for you to deal with large amounts of data to stress test your models.
- eli_gottlieb 8y ago
  >Bayesian can be seen as a subset of deep learning or hell a superset. eh-hem DIE, HERETIC! eh-hem Ok, with that out of my system, no, Bayesian methods are definitely not a subset of deep learning, in any way. Hierarchical Bayes could be labeled "deep Bayesian methods" if we're marketing jerks, but Bayesian methods mostly do not involve neural networks with >3 hidden layers. It's just a different paradigm of statistics.
  - digitalzombie 8y ago
    My mentor was very very adamant about Bayesian network and hierarchical as being deep learning. He sees the latent layer in the hierarchical model as the hidden layer and the Bayesian just have a strict restrictions/assumptions to the network where as the deep learning is more dumb and less assuming. A few of my professor thinks that PGM, probability graphical model is a super set of deep learning/neural network. This is where my thinking come from. IIRC, a paper have shown that gradient descent seems to exhibit MCMCs (blog with paper link inside that led to this conclusion of mine: http://www.inference.vc/everything-that-works-works-because-its-bayesian-2/ http://www.inference.vc/everything-that-works-works-because-...). But I am not an expert in Neural Network nor know the topic well enough to say such a thing. Other than was deferring to opinions of some one that's better than myself. So I'll keep this in mind and hopefully one day have the time to do more research into this topic. Thank you.
    - eli_gottlieb 8y ago
      I think your link, and your mentor, are somewhat fundamentalist about their Bayesianism.
Jach 8y ago
Honestly as much as it is slightly irritating to see deep learning hogging all the glory, there's a lot of money being sloshed around and quite a bit of it is spilling over to non-deep learning too. Which is great. An AI winter may be coming, though I think it's at minimum several years off, since big enterprises are just getting started with the most hyped things. If the hype doesn't return on its promises enough for sustained investment (that's a rather big if since the low hanging fruit aren't yet all picked) then the companies and funding will eventually recede, maybe even trigger another winter, but just as it takes a while to ramp up, it will also take a while to course correct. In the meantime all the related areas get better funding and attention (and chance to positively contribute to secure further investment) that they'd otherwise not have since we'd still be stuck in the low funding model from the last winter.
dx034 8y ago
I think the problem is the definition of AI. It appears most in the field define it as a superset of ML, encompassing all kinds of statistical methods and data analysis. For the general public, AI is a synonym for deep learning. When large companies speak about AI they always mean deep learning, never just a regression (probably also because many don't see a regression as intelligent). So AI in the public's perception could face a winter but much of the domain of machine learning would be unaffected.
- xamuel 8y ago
  >For the general public, AI is a synonym for deep learning I'd contend for the general public, AI is a synonym for machines like: HAL; The Terminator; Star Trek's "Data"; the robots in the film "AI"; and so on. We're nowhere remotely in the vicinity of that, and no-one even has any plausible ideas about how to start. A random person outside of tech probably doesn't even know what deep learning is. They might have heard of it somewhere in passing.

by imh 8y ago

digitalzombie 8y ago

Bayesian can be seen as a subset of deep learning or hell a superset. AI is a superset and Machine learning is a subset of AI and most funding is in deep learning. Once Deep Learning hit the limit I believe there will be an AI winter. Maybe there will be hype around statistic (cross fingers) which will lead to Bayesian and such.

johnmoberg 8y ago
How can Bayesian stuff be seen as a subset or superset of deep learning?
- kamaal 8y ago
  I guess the point that digitalzombie is trying to make is most of what we call AI or ML or even Deep learning is simply extension of statistics on computers. Things like the German tank problem or the problem of hardening airplanes during WW2 have that very AI'esque feel to it. Where you use data to build a model, then let that data from the model to change the model as it fits. Also the whole thing about 'decision making' is either bayesian or frequency based models in nature. Most of these algorithms and math has long existed before the current boom. Its just that the raw computing power and resources that you have today make it possible for you to deal with large amounts of data to stress test your models.
eli_gottlieb 8y ago
>Bayesian can be seen as a subset of deep learning or hell a superset. eh-hem DIE, HERETIC! eh-hem Ok, with that out of my system, no, Bayesian methods are definitely not a subset of deep learning, in any way. Hierarchical Bayes could be labeled "deep Bayesian methods" if we're marketing jerks, but Bayesian methods mostly do not involve neural networks with >3 hidden layers. It's just a different paradigm of statistics.
- digitalzombie 8y ago
  My mentor was very very adamant about Bayesian network and hierarchical as being deep learning. He sees the latent layer in the hierarchical model as the hidden layer and the Bayesian just have a strict restrictions/assumptions to the network where as the deep learning is more dumb and less assuming. A few of my professor thinks that PGM, probability graphical model is a super set of deep learning/neural network. This is where my thinking come from. IIRC, a paper have shown that gradient descent seems to exhibit MCMCs (blog with paper link inside that led to this conclusion of mine: http://www.inference.vc/everything-that-works-works-because-its-bayesian-2/ http://www.inference.vc/everything-that-works-works-because-...). But I am not an expert in Neural Network nor know the topic well enough to say such a thing. Other than was deferring to opinions of some one that's better than myself. So I'll keep this in mind and hopefully one day have the time to do more research into this topic. Thank you.
  - eli_gottlieb 8y ago
    I think your link, and your mentor, are somewhat fundamentalist about their Bayesianism.

Jach 8y ago

Honestly as much as it is slightly irritating to see deep learning hogging all the glory, there's a lot of money being sloshed around and quite a bit of it is spilling over to non-deep learning too. Which is great. An AI winter may be coming, though I think it's at minimum several years off, since big enterprises are just getting started with the most hyped things. If the hype doesn't return on its promises enough for sustained investment (that's a rather big if since the low hanging fruit aren't yet all picked) then the companies and funding will eventually recede, maybe even trigger another winter, but just as it takes a while to ramp up, it will also take a while to course correct. In the meantime all the related areas get better funding and attention (and chance to positively contribute to secure further investment) that they'd otherwise not have since we'd still be stuck in the low funding model from the last winter.

dx034 8y ago

I think the problem is the definition of AI. It appears most in the field define it as a superset of ML, encompassing all kinds of statistical methods and data analysis. For the general public, AI is a synonym for deep learning. When large companies speak about AI they always mean deep learning, never just a regression (probably also because many don't see a regression as intelligent). So AI in the public's perception could face a winter but much of the domain of machine learning would be unaffected.

xamuel 8y ago
>For the general public, AI is a synonym for deep learning I'd contend for the general public, AI is a synonym for machines like: HAL; The Terminator; Star Trek's "Data"; the robots in the film "AI"; and so on. We're nowhere remotely in the vicinity of that, and no-one even has any plausible ideas about how to start. A random person outside of tech probably doesn't even know what deep learning is. They might have heard of it somewhere in passing.