4 ms·
Weird I was just looking at this last night. They implemented it: https://github.com/ZigaSajovic/dCpp https://github.com/ZigaSajovic/dCpp I got onto that base
by ilaksh 10y ago
Weird I was just looking at this last night. They implemented it: https://github.com/ZigaSajovic/dCpp https://github.com/ZigaSajovic/dCpp
I got onto that based on this fantasy I was having. Basically I keep thinking that the main advantage deep learning has is probably not actually the neural nets, but the fact that they are highly parallel and very general computation techniques.
So if it doesn't really have to be a NN, maybe there could be a way to more efficiently achieve the same level of results by using something more like evolved parallel programs. Maybe have a universal virtual sensor grounding. Then if you find a useful/general procedure/dataset you could distribute it to improve lots of diverse agent's knowledge/skills immediately.
But to evolve these supposed universal function-approximating parallel programs that are more efficient and exchangable than deep NNs, you would need something like backpropagation, except for some arbitrary program structures. So when reading about backpropagation on wikipedia they mention the generalization is https://en.m.wikipedia.org/wiki/Automatic_differentiation https://en.m.wikipedia.org/wiki/Automatic_differentiation
Anyway I don't grock all the details fully but my interpretation of all of this is that I think things like backpropagation with deep NNs are truly general purpose and when combined with grounding such as streams of virtual/real senses and training across widely varied tasks can actually produce general human and animal/like intelligence. I think the problem is it just takes a ton of computing power, iterations and time. If we wait 2-10 years that probably won't matter. But if we don't want to wait we can probably do something similar and more efficient/readily distributable if we can use algorithms that aren't NNs but are general and highly parallel and can be adjusted via AD to learn.
Based on this I personally believe we will see human/animal like general intelligence systems before 2020.
- gallerdude 10y agoSo, we could use something like OpenAI Universe to train such generally intelligent systems?
- ilaksh 10y agoYes I think there are multiple serious AGI efforts using something like OpenAI Universe (there are others like that) and even though I haven't seen any earth-shattering announcements yet, I think between the AD stuff and hybrid data structure/program/NN systems and all of the investment in more powerful NN hardware and configurations there will be some surprisingly general demos running in open/varied environments in the next year or two.
- argonaut 10y agoI don't think there are any serious AGI efforts using OpenAI Universe.
- argonaut 10y agoYou have to be a little careful here. Neural networks are not "very general computation techniques." A dot product and a rectified linear function (or some other function of choice) are not "general computation techniques" in the sense you seem to use. They are a very specific set of operations. And the fact that you can show that two layers of these operations is a universal approximator is a red herring: decision trees and k nearest neighbors are also universal approximators.
- ilaksh 10y agoSure but I thought it was clear that I wasn't speaking in a precise or mathematical way. I was referring to what seems to be amazingly general abilities of certain successful deep learning-ish systems. And potentially also possibly amazingly general capabilities of some supposed imagined system a little bit like like those in terms the AD versus backpropagation. Do you really think these abilities are not or cannot be as general as I imagine? I just see these systems like the Deep Mind projects reading pixels in 2d and 3d, and having this low-level sensory grounding. It seems likely to me that we will see very exciting and general further results and then down the line similar things on perhaps much smaller computers with the non-NN efficiencies. Do you think this is impossible or unlikely?
- argonaut 10y agoIf you cannot describe something in a way that would allow for a precise formulation, then I don't know what you're trying to say. This is the difference between something that is implementable / testable, and something that's just a fantasy. As far as I can tell, your definition of "general computing" is: "it works, so it must be general." While this is ironically indeed the attitude of a lot of people in deep learning, this is just faulty logic.
- Smaug123 10y ago"I personally believe we will see human/animal like general intelligence systems before 2020." What odds would you give me for that? If I agreed to pay you $100 in the event that we created human/animal-like general intelligence systems before 2020, how much would you agree to pay me in the event that we fail to do so?
- ilaksh 10y agoThe problem is that there are a lot of things I would consider to be 'human/animal-like general intelligence' that you probably wouldn't.. especially if it came down to money. But for my purposes it would be something that could maybe get hints from virtual signs with simple words or phrases and then learn to do novel tasks in multiple complex 3d simulated environments where to some degree some learning was transferred. And then if it could also interact with virtual web pages on screens in those 3d worlds that had realistic use cases such as input forms, captchas, etc. Maybe we could even speak to it using language that a 3 year old would understand. I think by the end of 2019 this is quite likely. So say 2-to-1 odds. So if I win, I get $50, if you win, you get $100. But whether its a few years or a few decades, I truly believe that the human era is coming to an end. As in, regular people that don't have some sort of high-bandwidth brain interface to a contemporary AI, within not too many years, will be mainly irrelevant in the same way that many would consider chimpanzees to be irrelevant. At that point they will be technically transhumans, but the main drivers for our world will likely not be tied to human brains. Exactly when it will get to that point I don't know and don't think I want to bet. But my guess is before 2027.
- ajkjk 10y agoIt blows my mind that you think the world could change that much in 10 years. I'd estimate the things you describe as happening maybe by 2100, if they're going to happen at all.
- taneq 10y agoAs I understand it, the motivation behind DeepMind's "differentiable neural computer" work is to build something Turing-complete but differentiable. The "neural" part isn't intrinsic to the goal, it's just the most promising way we currently have to get there. A non-neural-net-based differentiable computer would be just as good, possibly better since the proposed neural architectures seem pretty inefficient.
- Frogolocalypse 10y agoActually, I've been playing with this stuff as a hobby for about 15 years. There was a book by Stephen Levy i remember reading that kind of kicked it off for me.
- marcosdumay 10y agoI was sure there was a generic term for that (at least, some generic term more specific than AI), but I can only specific ones. Those would be interesting: https://en.wikipedia.org/wiki/Genetic_programming https://en.wikipedia.org/wiki/Genetic_programming https://en.wikipedia.org/wiki/Inductive_programming https://en.wikipedia.org/wiki/Inductive_programming Keep in mind that CPU time is a huge bottleneck. Genetic algorithms are easy to parallelize, while inductive searches are less so, also inductive searches are not cache friendly. Still, if you try them, you'll see that what I said about CPU time is a huge understating. Turns out that the search space of all programs in a given language is big.