9 ms·

AI winters are a result of a massive disparity between the expectations of the general public and the reality of where the technology currently sits. Just like

AI winters are a result of a massive disparity between the expectations of the general public and the reality of where the technology currently sits. Just like an asset bubble, the value of the industry as a whole pops as people collectively realize that AI, while not being worthless, is worth significantly less than they thought. Understand that in pop-sci circles over the past several years the general public is being exposed to stories warning about the singularity by well respected people like Stephen Hawking and Elon Musk (http://time.com/3614349/artificial-intelligence-singularity-stephen-hawking-elon-musk/ http://time.com/3614349/artificial-intelligence-singularity-...). Autonomous vehicles are on the roads and Boston Dynamics is showing very real robot demonstrations. Deep learning is breaking records in what we thought was possible with machine learning. All of this progress has excited an irrational exuberance in the general public. But people don't have a good concept of what these technologies can't do, mainly because researchers, business people, and journalists don't want to tell them--they want the money and attention. But eventually the general public wises up to the unfulfillment of expectations, and drives their attention elsewhere. Here we have the AI winter.

varelse 8y ago
I'd clarify that there is a specific delusion that any data scientist straight out of some sort of online degree program can go toe to toe with the likes of Andrej Karpathy or David Silver with the power of "teh durp lurnins'." And the predictable disappointment arising from the craptastic shovelware they create is what's finally creating the long overdue disappointment. Further, I have repeatedly heard people who should know better, with very fancy advanced degrees, chant variants of "Deep Learning gets better with more data" and/or "Deep Learning makes feature engineering obsolete" as if they are trying to convince everyone around them as well as themselves that these two fallacious assumptions are the revealed truth handed down to mere mortals by the 4 horsemen of the field. That said, if you put your ~10,000 hours into this, and keep up with the field, it's pretty impressive what high-dimensional classification and regression can do. Judea Pearl concurs: https://www.theatlantic.com/technology/archive/2018/05/machine-learning-is-stuck-on-asking-why/560675/ https://www.theatlantic.com/technology/archive/2018/05/machi... My personal (and admittedly biased) belief is that if you combine DL with GOFAI and/or simulation, you can indeed work magic. AlphaZero is strong evidence of that, no? And the author of the article in this thread is apparently attempting to do the same sort of thing for self-driving cars. I wouldn't call this part of the field irrational exuberance, I'd call it amazing.
- bitL 8y ago
  > Deep Learning makes feature engineering obsolete I think even if you avoid constructing features, you are basically doing a similar process where a single change in a hyper-parameter can have significant effects: - internal structure of a model (what types of blocks are you using and how do you connect them, what are they capable of together, how do gradients propagate?) - loss function (great results come only if you use a fitting loss function) - category weights (i.e. improving under-represented classes) - image/data augmentation (self-driving car won't work without significant augmentation at all) - properly set-up optimizer The good thing here is that you can automate optimization of these to a large extent if you have a cluster of machines and a way to orchestrate meta-optimization of slightly changed models. With feature engineering you just have to do all the work upfront, thinking what might be important, and often you just miss important parts of features :-(
  - varelse 8y ago
    Yep, and in doing so, you just traded "feature engineering" for graph design and data prep, no? And that's my response to these sorts. And their usual response to me is to grumble that I don't know what I'm doing. I've started tuning them out of my existence because they seem to have nothing to contribute to it.
    - jacquesm 8y ago
      It's a huge difference in terms of the time invested to create something that performs well. Hand crafted feature engineering is better for some tasks but for quite a few of them automated methods perform very well indeed (at least, better than I expected).
  - riku_iki 8y ago
    > if you have a cluster of machines and a way to orchestrate meta-optimization of slightly changed models curious, if there is any good quality open source project for this..
    - 8y ago
kinsomo 8y ago
> But eventually the general public wises up to the unfulfillment of expectations, and drives their attention elsewhere. Here we have the AI winter. And more importantly, business and government leaders wise up and turn off the money tap.
- khawkins 8y ago
  This is why it's wise for researchers and business leaders to temper expectations. Better a constant modest flow of money into the field than a boom-bust cycle with huge upfront investment followed by very bearish actions.
  - pasabagi 8y ago
    I think the problem is, that's absolutely against the specific interests of university departments, individual researchers, and newspapers - even if it's in the interest of the field as a whole.
    - sram1337 8y ago
      Prisoner's Dilemma
  - kabouseng 8y ago
    That requires super rational agents in the game theory sense...
Florin_Andrei 8y ago
> AI winters are a result of a massive disparity between the expectations of the general public and the reality of where the technology currently sits. I think they also happen when the best ideas in the field run into the brick wall of insufficiently developed computer technology. I remember writing code for a perceptron in the '90s on an 8 bit system, 64 k RAM - it's laughable. But right now compute power and data storage seem plentiful, so rumors of the current wave's demise appear exaggerated.
- bitL 8y ago
  I wish GPUs were 1000x faster... Then I could do some crazy magic with Deep Learning instead of waiting weeks for training to be finished...
  - pizza 8y ago
    somebody is working on photonics-based ML http://www.lighton.io/our-technology http://www.lighton.io/our-technology
  - jacquesm 8y ago
    That's more a matter of budget than anything else. If you problem is valuable enough spending the money in a short time-frame rather than waiting for weeks can be well worth the investment.
    - bitL 8y ago
      I cannot fit a cluster of GPUs into a phone where I could make magic happen real-time though :(
      - jacquesm 8y ago
        Hm. Offload the job to a remote cluster? Or is comms then the limiting factor?
        8y ago
cyberpunk0 8y ago
> AI winters are a result of a massive disparity between the expectations of the general public and the reality of where the technology currently sits. A symptom of capitalism and marketing trying to push shit they don't understand

varelse 8y ago

I'd clarify that there is a specific delusion that any data scientist straight out of some sort of online degree program can go toe to toe with the likes of Andrej Karpathy or David Silver with the power of "teh durp lurnins'." And the predictable disappointment arising from the craptastic shovelware they create is what's finally creating the long overdue disappointment. Further, I have repeatedly heard people who should know better, with very fancy advanced degrees, chant variants of "Deep Learning gets better with more data" and/or "Deep Learning makes feature engineering obsolete" as if they are trying to convince everyone around them as well as themselves that these two fallacious assumptions are the revealed truth handed down to mere mortals by the 4 horsemen of the field. That said, if you put your ~10,000 hours into this, and keep up with the field, it's pretty impressive what high-dimensional classification and regression can do. Judea Pearl concurs: https://www.theatlantic.com/technology/archive/2018/05/machine-learning-is-stuck-on-asking-why/560675/ https://www.theatlantic.com/technology/archive/2018/05/machi... My personal (and admittedly biased) belief is that if you combine DL with GOFAI and/or simulation, you can indeed work magic. AlphaZero is strong evidence of that, no? And the author of the article in this thread is apparently attempting to do the same sort of thing for self-driving cars. I wouldn't call this part of the field irrational exuberance, I'd call it amazing.

bitL 8y ago
> Deep Learning makes feature engineering obsolete I think even if you avoid constructing features, you are basically doing a similar process where a single change in a hyper-parameter can have significant effects: - internal structure of a model (what types of blocks are you using and how do you connect them, what are they capable of together, how do gradients propagate?) - loss function (great results come only if you use a fitting loss function) - category weights (i.e. improving under-represented classes) - image/data augmentation (self-driving car won't work without significant augmentation at all) - properly set-up optimizer The good thing here is that you can automate optimization of these to a large extent if you have a cluster of machines and a way to orchestrate meta-optimization of slightly changed models. With feature engineering you just have to do all the work upfront, thinking what might be important, and often you just miss important parts of features :-(
- varelse 8y ago
  Yep, and in doing so, you just traded "feature engineering" for graph design and data prep, no? And that's my response to these sorts. And their usual response to me is to grumble that I don't know what I'm doing. I've started tuning them out of my existence because they seem to have nothing to contribute to it.
  - jacquesm 8y ago
    It's a huge difference in terms of the time invested to create something that performs well. Hand crafted feature engineering is better for some tasks but for quite a few of them automated methods perform very well indeed (at least, better than I expected).
- riku_iki 8y ago
  > if you have a cluster of machines and a way to orchestrate meta-optimization of slightly changed models curious, if there is any good quality open source project for this..
  - 8y ago

kinsomo 8y ago

> But eventually the general public wises up to the unfulfillment of expectations, and drives their attention elsewhere. Here we have the AI winter. And more importantly, business and government leaders wise up and turn off the money tap.

khawkins 8y ago
This is why it's wise for researchers and business leaders to temper expectations. Better a constant modest flow of money into the field than a boom-bust cycle with huge upfront investment followed by very bearish actions.
- pasabagi 8y ago
  I think the problem is, that's absolutely against the specific interests of university departments, individual researchers, and newspapers - even if it's in the interest of the field as a whole.
  - sram1337 8y ago
    Prisoner's Dilemma
- kabouseng 8y ago
  That requires super rational agents in the game theory sense...

Florin_Andrei 8y ago

> AI winters are a result of a massive disparity between the expectations of the general public and the reality of where the technology currently sits. I think they also happen when the best ideas in the field run into the brick wall of insufficiently developed computer technology. I remember writing code for a perceptron in the '90s on an 8 bit system, 64 k RAM - it's laughable. But right now compute power and data storage seem plentiful, so rumors of the current wave's demise appear exaggerated.

bitL 8y ago
I wish GPUs were 1000x faster... Then I could do some crazy magic with Deep Learning instead of waiting weeks for training to be finished...
- pizza 8y ago
  somebody is working on photonics-based ML http://www.lighton.io/our-technology http://www.lighton.io/our-technology
- jacquesm 8y ago
  That's more a matter of budget than anything else. If you problem is valuable enough spending the money in a short time-frame rather than waiting for weeks can be well worth the investment.
  - bitL 8y ago
    I cannot fit a cluster of GPUs into a phone where I could make magic happen real-time though :(
    - jacquesm 8y ago
      Hm. Offload the job to a remote cluster? Or is comms then the limiting factor?
      - 8y ago

cyberpunk0 8y ago

> AI winters are a result of a massive disparity between the expectations of the general public and the reality of where the technology currently sits. A symptom of capitalism and marketing trying to push shit they don't understand