5 ms·
I understand and empathize with the skepticism or rather criticisms around hand wringing with respect to the implications of current deep learning methods. How
by AndrewKemendo 10y ago
I understand and empathize with the skepticism or rather criticisms around hand wringing with respect to the implications of current deep learning methods.
However, as someone who builds them for vision applications I'm increasingly convinced that some form of ANN will underlie AGI - what he calls a universal algorithm.
If we assume that general intelligence comes from highly trained, highly connected single processors (neurons) with a massive and complex sensor system, then replicating that neuron is step one - which arguably is what we are building, albeit comparatively crudely, with ANN's.
If you compare at a high level how infants learn and how we train RNN/CNNs they are remarkably similar.
I think where the author, and in general the ML crowd focuses too much is on unsupervised learning as being pivotal for AGI.
In fact if you look again at biological models the bulk of animal learning is supervised training in the strict technical sense. Just look at feral children studies as proof of this.
Where the author detours too much is assuming the academic world would prove a broader scope for ANN if it were there. In fact however research priorities are across the board not focused on general intelligence and most machine learning programs explicitly forbid this research for graduate students as it's not productive over the timeline of a program.
Bengio and others I think are on the right track, focusing on the question of ANN towards AGI and I think it will start producing results as our training methods.
- clmcleod 10y agoHey Andrew, First, I'm curious where you thought I was focused on unsupervised learning? It certainly didn't cross my mind when I was writing this --- I was (implicitly) strictly talking about supervised machine learning. My post actually is in support of what the latter of your comment says in a round-a-bout way. In general, the people that are making huge strides in deep learning (Bengio, Hinton, Lecun are obviously the big three) understand the capabilities and, maybe more importantly, the limitations of DL. My main point is that the ML community at large is actually not on the same page as the experts, and that causes many more problems. I want us, as a community, to stop treating deep learning any different than any other ML algorithms --- have a consensus, based on scientific facts, about the possibilities and limitations thus-far. If we, "the experts", don't understand these things about our own algorithms, how can we the rest of the world to understand them?
- dave_sullivan 10y ago> I want us, as a community, to stop treating deep learning any different than any other ML algorithms --- have a consensus, based on scientific facts, about the possibilities and limitations thus-far. If we, "the experts", don't understand these things about our own algorithms, how can we the rest of the world to understand them? I agree. It's interesting watching the "debate" around deep learning. All the empirical results are available for free online, yet there's so much misinformation and confusion. If you're familiar with the work, you can fill in the blanks on where things are headed. For instance, in 2011, I think it became clear that RNNs were going to become a big thing, based on work from Ilya Sutskever and James Martens. Ilya was then doing his PhD and is now running OpenAI, doing research backed by a billion dollars. The pace of change in deep learning is accelerating. It used to be fairly easy for me to stay current with new papers that were coming out; now I have a backlog. To a certain extent, it doesn't matter what other people think, much of the debate is just noise. I don't know what AGI is. If it's passing the turing test, we're pretty close, 3 years max, maybe by the end of the year. Anything more than that is too metaphysical and prone to interpretation. But there have been a bunch of benchmark datasets/tasks established now. Imagenet was the first one that everyone heard about I think, but sets like COCO, 1B words, and others have come out since then and established benchmarks. Those benchmarks will keep improving, pursuing those improvements will lead to new discoveries re: "intelligence as computation", and something approximately based on "deep learning" will drive it for a while.
- deleted 10y ago[deleted]
- gambler 10y ago> If it's passing the turing test, we're pretty close, 3 years max Well yes? If a Turing test you realize the simulation of some idiot in the online chat, it has long been there - and nobody wants. But the system, which can lead a meaningful conversation, today there is no trace. And there is even no harbingers of its occurrence. <- This was translated by Google Translate from a piece of perfectly intelligible and grammatically correct text in another language. If this is the state of the art in machine translation, how on Earth can you expect a machine that can converse on human level in three years?
- thaumasiotes 10y ago> In fact if you look again at biological models the bulk of animal learning is supervised training in the strict technical sense. Just look at feral children studies as proof of this. Feral children are incapable of learning a language as adults. But children don't require input (or feedback) from an existing language in order to "learn" a language; a community of children is capable of developing a full natural language without exposure to adult language. The problem the feral children encounter is not that no one teaches them to speak, it's that they're alone. This is much as if you gave a neural network "supervised training" by letting it interact with several other untrained copies of itself, but no coded data. Does that meet your definition of "supervised training"? (Obviously, a language created by an isolated group of children will be a new language rather than a known one. Children learning an existing language do need input, but there are some interesting quirks to the process -- they don't seem to need feedback, or use it if they get it, and in the natural course of things the input they get is 100% "correct" and 0% "incorrect". Try training a neural net to discriminate dogs from non-dogs when you can't code anything as "not a dog".)
- AndrewKemendo 10y agoDoes that meet your definition of "supervised training"? I think as a second layer, yes. To your original statement though, being alone is the same as no-one teaching them to talk. Adults, and other children, in fact teach infants to talk. It might not seem explicit (though there are very explicit programs for infant language development) but in fact it works the same way. See my other comment about object segmentation and classification.
- thaumasiotes 10y ago> To your original statement though, being alone is the same as no-one teaching them to talk. No it's not. You can have one without the other. Being alone implies no one teaching them to talk. However, children who are not alone still learn to talk despite no one teaching them.
- AndrewKemendo 10y ago
- otabdeveloper1 10y ago> If we assume that general intelligence comes from highly trained, highly connected single processors (neurons) This assumption is wrong, of course.
- AndrewKemendo 10y agoDid you have anything more to add to that, like any pointing toward something else, or was it just internet snark?
- eli_gottlieb 10y ago>If you compare at a high level how infants learn and how we train RNN/CNNs they are remarkably similar. That doesn't sound correct. Infants engage in active inference to resolve their uncertainties, engage in transfer learning between different domains by default, and use their transfer-learning abilities to learn new categories or domains from even just one example.
- AndrewKemendo 10y agoSo I would say - kind of. But at it's root, the mechanisms are based on having a social network of people touching, talking, playing etc... that does the training - though it might not look the same. A perfect example I use all the time which is much later in development but relevant, is when a 1.5 year old will point at something and ask if it's a [thing]. The initial training was people segmenting (holding, moving different ways) classifying (calling it [thing], asking for [thing], telling the subject it is [thing]), which was then reinforced/weighted by asking if [thing] was indeed [thing]. So it doesn't look like explicit training, but it follows all the same steps.