3 ms·

Imo gradient descent is not going to get us to general AI. I'm a programming languages theory person, and I also know a fair bit about machine learning. The cur

by calebh 8y ago

Imo gradient descent is not going to get us to general AI. I'm a programming languages theory person, and I also know a fair bit about machine learning. The current problem with neural networks is that they suck at processing variable length data, and they have problems with remembering the past. Programs can be represented by trees, graphs, or text. Deep learning isn't great at dealing with graphs or trees, and it's not that good at text either. The other issue is that deep learning works great for recognizing common patterns, but it sucks when faced with novel situations. So I don't think that we're going to have programs that program anytime soon. The first applications of AI to programming will probably be with programming assistants or with AI guided proof solvers. Programmer productivity will improve, but we're not going to see everyone losing their jobs.

nmca 8y ago
The latter of those two points is much more valid than the former. You're comment about generalisation difficulty is pretty accurate, but as for "sucks at variable length data", I think neural machine translation [0] and the fact that schemes including RNNs just won M4 [1] indicate that this is incorrect. Your point about remembering the past is true (it's hard), but people are actively working on it. Unitary neural nets, and the fast/slow weight paradigm are very different angles that seem promising. As for handling trees + graphs, this actually works very well. Thomas Kipf is pushing this area forward, GATs [2] are a nice random example of how dominant differentiable programming (eg NNs) can be in this area. Unfortunately these graph approaches don't parallelise as nicely on GPUs as CNNs. Your predictions (assistants) seem likely to me. [0] https://arxiv.org/abs/1804.09849 https://arxiv.org/abs/1804.09849 [1] https://www.m4.unic.ac.cy https://www.m4.unic.ac.cy [2] https://arxiv.org/abs/1710.10903 https://arxiv.org/abs/1710.10903

by calebh 8y ago

nmca 8y ago

The latter of those two points is much more valid than the former. You're comment about generalisation difficulty is pretty accurate, but as for "sucks at variable length data", I think neural machine translation [0] and the fact that schemes including RNNs just won M4 [1] indicate that this is incorrect. Your point about remembering the past is true (it's hard), but people are actively working on it. Unitary neural nets, and the fast/slow weight paradigm are very different angles that seem promising. As for handling trees + graphs, this actually works very well. Thomas Kipf is pushing this area forward, GATs [2] are a nice random example of how dominant differentiable programming (eg NNs) can be in this area. Unfortunately these graph approaches don't parallelise as nicely on GPUs as CNNs. Your predictions (assistants) seem likely to me. [0] https://arxiv.org/abs/1804.09849 https://arxiv.org/abs/1804.09849 [1] https://www.m4.unic.ac.cy https://www.m4.unic.ac.cy [2] https://arxiv.org/abs/1710.10903 https://arxiv.org/abs/1710.10903