3 ms·
I've been spending a lot of time lately digging into "outdated" 80's era approaches to AI. Some people think I'm wasting my time and ask "why aren't you doing
by mindcrime 8y ago
I've been spending a lot of time lately digging into "outdated" 80's era approaches to AI. Some people think I'm wasting my time and ask "why aren't you doing Deep Learning?" This is the reason why:
"You can think about deep learning as it currently is today as the equivalent in the brain to our sensory cortices: our visual cortex or auditory cortex. But, of course, true intelligence is a lot more than just that, you have to recombine it into higher-level thinking and symbolic reasoning, a lot of the things classical AI tried to deal with in the 80s. One way you can think about our research program is [that it's investigating] 'Can we build out from our perception, using deep-learning systems and learning from first principles? Can we build out all the way to high-level thinking and symbolic thinking?'. In order to do that we need to crack problems like learning concepts, things that humans find effortless but our current learning systems can't do."
I basically completely agree with Demis on this. A lot of the stuff that was being done in the 80's wasn't "wrong" so much as it was ahead of its time. And just like advances in data, algorithms, and compute power helped propel Neural Networks (which are actually a really old idea) to another level, I suspect some other "old" approaches will also experience a similar resurgence due to that same confluence of factors.
- dmreedy 8y agoYes, I like to separate, such as they can be separated, the tasks of 'parsing' and 'thinking' when it comes to AI. In the 80s (and really, all the way back to the 50s when we started trying computational AI), the tasks that we were attempting to solve were in the `thinking` category; solving tasks at a high level of abstraction with complex ontologies. The revolution in Deep Learning has been almost entirely on the parsing side of things. The conversion of unstructured data from sensors (an array of pixels, say) into semi-structured semantic units in the beginnings of an ontology. This makes sense, because some of the greatest success has come from modeling a part of the brain that we understand pretty well; the occipital lobe (specifically, V1), which is responsible for parsing the input from the eyes and feeding the resulting information in to the rest of the brain. This is the motivation behind the layered, columned CNN architecture. We've seen some success performing some tasks that exist in the (quite large) grey space between parsing and thinking (which are probably platonic ideals anyway, more than they are real things), but I get the feeling we're reaching the limit of how far we can reach into the grey starting from the left side of this continuum. So I don't think it's a waste of time at all to consider the deep, impressive work from the 80s and the 60s in Philosophy and logic and GOFAI.
- YeGoblynQueenne 8y agoThat particular comment is very likely traceable to some work undertaken by DeepMind in learning symbolic representations using deep learning, described here: Learning explanatory rules from noisy data https://deepmind.com/blog/learning-explanatory-rules-noisy-data/ https://deepmind.com/blog/learning-explanatory-rules-noisy-d... From where I'm standing (studying symbolic machine learning algorithms) the obvious next step in machine learning and AI is the combination of sub-symbolic, with symbolic approaches. It's just, nobody has quite figured out how to make it all work yet. There are reasons for this, not all technical, but it seems to me it's the easiest "next big thing" to at least try.
- mindcrime 8y agoFrom where I'm standing (studying symbolic machine learning algorithms) the obvious next step in machine learning and AI is the combination of sub-symbolic, with symbolic approaches. It's just, nobody has quite figured out how to make it all work yet. There are reasons for this, not all technical, but it seems to me it's the easiest "next big thing" to at least try. I 100% agree, FWIW. I believe that figuring out how to properly integrate those different regimes will be a Big Deal.
- ableal 8y ago> digging into "outdated" 80's era I'd say that's a good idea. A 30-year cycle (say one human generation) seems to occur in many fields, for various reasons. For instance, some of the first transistors were field-effect (FETs), then bipolar junction (BJTs) ruled the first integrated circuits, then MOS technology made FETs again the standard technology. Some of the design approaches were recycled ... Sometimes things seem to go in a spiral, and the new level amplifies the previous. I'd just caution that knowing why the previous approach went out of fashion would be useful, but that's often hard to determine - acknowledging failure is not a popular activity. Lack of power may not be the only problem ...
- EGreg 8y agoI have one question for you and Demis What if logic itself is just reduction to a simple binary system with few moving parts - the easy case? Being able to logically explain how something works in special cases is what we have been able to do until now with pencil and paper. Then you get approaches in the more general case, like New Kind of Science by Stephen Wolfram to “study” self-similarity. And you have fluid dynamics, which oncd again tries to find analytic closed-form solutions after averaging. But what if - with computers - you can GENERALIZE REASONING ABOUT MODELS to something far beyond logical explanation? And it’s not reducible to a few main factors, but instead relies on precomputing HUGE VECTORS via gradient descent or MCTS (like AlphaZero) and store that information compactly. So you can’t really explain “why” this move is (one of) the best moves, it just is. What if you could generalize AND handle a greater set of problems and “logic” was just a subset of it after all? Ideas like “all swans are white” and syllogisms are just approximations of the real world. With deep learning you can find patterns that contain far more variables and complex interrlationships. So maybe the 80s AI will suddenly emerge as a super-logical reasoning computer, not from efforts like Cyc which attempt to feed it data, but from general purpose deep learning with zero experience! What if it is suddenly better than everything humanity has ever produced or reasoned about, similar to how AlphaZero beat every bespoke chess computer program ever written after a few hours of training?
- mindcrime 8y agoSo maybe the 80s AI will suddenly emerge as a super-logical reasoning computer, not from efforts like Cyc which attempt to feed it data, but from general purpose deep learning with zero experience! That's a fair point, and I've spent some time thinking about roughly the same thing. My current thought on it is this: "general intelligence" may well turn out to be an emergent property of very simple lower-level primitives, and it may be that what we call "deep learning", done with a sufficiently deep and wide network, might actually produce "general intelligence". But... I feel like we're still pretty far from having the compute power to run an ANN of such scale (that's just a guess though). So where we can "help" the process by hand-engineering modules that do specific elements of what we call intelligence, combine those modules, that's one way to short-cut the process. And in my view, things like Deep Learning, DRL, MCTS, etc. would also be included, along with older approaches: PCT for abductive inference, logical inference, maybe something like symbolic regression with GA's, etc., etc.
- qbilius 8y agoWhat would be a good starting point for digging into that 80s stuff, in your opinion?
- mindcrime 8y agoGood question. There's a LOT of material out there, and it could take a lifetime to parse through it all. So my own approach has been to skim a lot and latch onto whatever superficially seems interesting at first blush, do a deeper dive, and decide whether or not to keep digging. As it happens, one of the first things I latched onto was abductive inference, and an approach to automating same called "parsimonious covering theory". I've been working on my own implementation of PCT as a way to learn that. I also think there are some interesting ideas in Minsky's The Society of Mind approach, so I've been going through some of his stuff. That and the old "Blackboard Architecture" model, which I see as having a tenuous sort of connection. And while it's more "90's" than "80's", I've also been reading a lot of older literature on Multi-Agent Systems lately. There's been a lot of talk lately about "Multi-Agent Reinforcement Learning" so I'm interested in that entire space. Really, there's a thin, tenuous thread connecting most of my interests in this area (with the PCT stuff being sort of un-connected w/r/t the rest)... MAS's, Blackboard Architecture, Society of Mind, all deal with the idea of multiple "things" (call them modules, agents, components, whatever) working together in some fashion. I suspect there's "some there there" in terms of unifying all of these various AI approaches and achieving something useful. I could be wrong, but this is where I've been directing a lot of energy lately.
- abecedarius 8y agoRussell & Norvig devote considerable space to it, and their chapters all end with a great historical review section.