5 ms·

>What is new is the hype around deep learning that took off after 2012, and because Google and Facebook decided to champion it. Have you seen what neural nets

>What is new is the hype around deep learning that took off after 2012, and because Google and Facebook decided to champion it. Have you seen what neural nets are now capable of? Speech synthesis/transcription, voice synthesis, image synthesis/labeling/infill, style transfer, music synthesis, and a host of other classes of optimization problems which have intractable explicit programmitic solutions. The hype is justified, because ML has finally arrived, thanks primarily to hardware, and secondarily to the wealth of modern open research, heavily influenced congregations of leading researchers enabled by funding at Google, Facebook, etc. The problems being solved by "curve fitting" ML were simply unsolvable by any practical, generalizable means before recently, and the revolution is just getting started.

YeGoblynQueenne 7y ago
Yes, I've seen what neural nets are now capable of- they are capable of exactly what they were always capable of, except "now" (in the last few years) we have more data and more compute to train them to actually do it. Says Geoff Hinton [1]. I have also seen what neural nets are incapable of. Specifically, generalisation and reasoning. Says François Chollet of Keras [2]. AI, i.e. the sub-field of computer science research that is called "AI" and that consists of conferences such as AAAI, IJCAI, NeurIPS, etc, and assorted journals, cannot progress on the back of a couple of neural net architectures incapable of generalisation and reasoning. We had reasoning down pat in the '80s. Eventually, the hype cycle will end, the Next Big Thing™ will come around and the hype cycle will start all over again. It's the nature of revolutions, see? So hold your horses. Deep learning is much more useful for AI researchers who want to publish a paper in one of the big AI conferences, and to the FANG companies who have huge data and compute, than it is to anyone else. Anyone else who wants to do AI will need to wait their turn and hope something else comes around that has reasonable requirements to use, and scales well. Just as the original article suggests. _________________ [1] http://techjaw.com/2015/06/07/geoffrey-hinton-deep-learning-in-baby-steps-and-the-future-of-google-and-ai/ http://techjaw.com/2015/06/07/geoffrey-hinton-deep-learning-... Geoffrey Hinton: I think it’s mainly because of the amount of computation and the amount of data now around but it’s also partly because there have been some technical improvements in the algorithms. Particularly in the algorithms for doing unsupervised learning where you’re not told what the right answer is but the main thing is the computation and the amount of data. [2] https://blog.keras.io/the-limitations-of-deep-learning.html https://blog.keras.io/the-limitations-of-deep-learning.html Say, for instance, that you could assemble a dataset of hundreds of thousands—even millions—of English language descriptions of the features of a software product, as written by a product manager, as well as the corresponding source code developed by a team of engineers to meet these requirements. Even with this data, you could not train a deep learning model to simply read a product description and generate the appropriate codebase. That's just one example among many. In general, anything that requires reasoning—like programming, or applying the scientific method—long-term planning, and algorithmic-like data manipulation, is out of reach for deep learning models, no matter how much data you throw at them. Even learning a sorting algorithm with a deep neural network is tremendously difficult.
- ScottBurson 7y ago
  > We had reasoning down pat in the '80s. We had something, but if we really had reasoning "down pat", we would not now be reading an article about someone faking automated app development. Programming is all about reasoning.
  - YeGoblynQueenne 7y ago
    That programming needs reasoning doesn't mean that reasoning is the only thing you need if you want to do (automated) programming, or that reasoning is only useful in (automated) programming. That is relevant to your comment. The work on automated reasoning (or "inference", etc) really started in the '50s with Church and Turing, then reached a peak in the late 80's and 90's with work on automated theorem proving (there was a great big push at the time to solve very hard problems to do with the soundness and completeness of inference procedures, particularly resolution) and is still going on (for example with Constraint Programming and Answer Set Programming etc). The result of this work was logic programming. I'm leaving out all the work on functional programming that was just another branch of the same tree, if you like, because I don't know it that well but I'm sure others on this board can complete the picture. Then of course there was all the other classical AI stuff on planning, grammar learning, game playing etc etc that you can read about in Russel & Norvig. Now, all this work could potentially be turned to the task of automated programming- but automated programming was never the goal of all that research. There was a lot of work on program synthesis, but that was just another AI sub-field with its own specific goals, that were not the overarching goals of the field as a whole. That is why we don't have automated programming at the push of a button, today: because it was never the main subject of AI research. Edit: bit of a plug. Like I say in another comment, my PhD is on algorithms that learn logic programs from examples and background knowledge (both of which are also logic programs). That's Inductive Logic Porgramming. Our stuff works. We can learn recursive programs and even invent sub-programs that are necessary to complete a programming task and that are not provided by the user. We are making big leaps all the time and we're way, way ahead of neural program synthesis and the like. There's also a whole field of Inductive Functional Programming that does the same stuff but with functional programming languages. Automating app development with that sort of technique is mainly a matter of engineering- the research is out there. But, you haven't heard anything about it because the hullaballoo about deep learning is covering everything else up and most people don't even know there is AI outside of deep learning. Hence my comments in this thread (rather obviously).

YeGoblynQueenne 7y ago

Yes, I've seen what neural nets are now capable of- they are capable of exactly what they were always capable of, except "now" (in the last few years) we have more data and more compute to train them to actually do it. Says Geoff Hinton [1]. I have also seen what neural nets are incapable of. Specifically, generalisation and reasoning. Says François Chollet of Keras [2]. AI, i.e. the sub-field of computer science research that is called "AI" and that consists of conferences such as AAAI, IJCAI, NeurIPS, etc, and assorted journals, cannot progress on the back of a couple of neural net architectures incapable of generalisation and reasoning. We had reasoning down pat in the '80s. Eventually, the hype cycle will end, the Next Big Thing™ will come around and the hype cycle will start all over again. It's the nature of revolutions, see? So hold your horses. Deep learning is much more useful for AI researchers who want to publish a paper in one of the big AI conferences, and to the FANG companies who have huge data and compute, than it is to anyone else. Anyone else who wants to do AI will need to wait their turn and hope something else comes around that has reasonable requirements to use, and scales well. Just as the original article suggests. _________________ [1] http://techjaw.com/2015/06/07/geoffrey-hinton-deep-learning-in-baby-steps-and-the-future-of-google-and-ai/ http://techjaw.com/2015/06/07/geoffrey-hinton-deep-learning-... Geoffrey Hinton: I think it’s mainly because of the amount of computation and the amount of data now around but it’s also partly because there have been some technical improvements in the algorithms. Particularly in the algorithms for doing unsupervised learning where you’re not told what the right answer is but the main thing is the computation and the amount of data. [2] https://blog.keras.io/the-limitations-of-deep-learning.html https://blog.keras.io/the-limitations-of-deep-learning.html Say, for instance, that you could assemble a dataset of hundreds of thousands—even millions—of English language descriptions of the features of a software product, as written by a product manager, as well as the corresponding source code developed by a team of engineers to meet these requirements. Even with this data, you could not train a deep learning model to simply read a product description and generate the appropriate codebase. That's just one example among many. In general, anything that requires reasoning—like programming, or applying the scientific method—long-term planning, and algorithmic-like data manipulation, is out of reach for deep learning models, no matter how much data you throw at them. Even learning a sorting algorithm with a deep neural network is tremendously difficult.

ScottBurson 7y ago
> We had reasoning down pat in the '80s. We had something, but if we really had reasoning "down pat", we would not now be reading an article about someone faking automated app development. Programming is all about reasoning.
- YeGoblynQueenne 7y ago
  That programming needs reasoning doesn't mean that reasoning is the only thing you need if you want to do (automated) programming, or that reasoning is only useful in (automated) programming. That is relevant to your comment. The work on automated reasoning (or "inference", etc) really started in the '50s with Church and Turing, then reached a peak in the late 80's and 90's with work on automated theorem proving (there was a great big push at the time to solve very hard problems to do with the soundness and completeness of inference procedures, particularly resolution) and is still going on (for example with Constraint Programming and Answer Set Programming etc). The result of this work was logic programming. I'm leaving out all the work on functional programming that was just another branch of the same tree, if you like, because I don't know it that well but I'm sure others on this board can complete the picture. Then of course there was all the other classical AI stuff on planning, grammar learning, game playing etc etc that you can read about in Russel & Norvig. Now, all this work could potentially be turned to the task of automated programming- but automated programming was never the goal of all that research. There was a lot of work on program synthesis, but that was just another AI sub-field with its own specific goals, that were not the overarching goals of the field as a whole. That is why we don't have automated programming at the push of a button, today: because it was never the main subject of AI research. Edit: bit of a plug. Like I say in another comment, my PhD is on algorithms that learn logic programs from examples and background knowledge (both of which are also logic programs). That's Inductive Logic Porgramming. Our stuff works. We can learn recursive programs and even invent sub-programs that are necessary to complete a programming task and that are not provided by the user. We are making big leaps all the time and we're way, way ahead of neural program synthesis and the like. There's also a whole field of Inductive Functional Programming that does the same stuff but with functional programming languages. Automating app development with that sort of technique is mainly a matter of engineering- the research is out there. But, you haven't heard anything about it because the hullaballoo about deep learning is covering everything else up and most people don't even know there is AI outside of deep learning. Hence my comments in this thread (rather obviously).