24 ms·

DeepMind says reinforcement learning is ‘enough’ to reach general AI

• ramraj07 5y ago
  Wait isn’t any Turing complete programming language sufficient to eventually reach general AI
  • randcraw 5y ago
    Like the notion that all of natural language can be modeled fully if you just use enough finite state machines — surely true; just a wee bit inefficient.
    • joe_the_user 5y ago
      If you had a custom chip it could be efficient. Determining which state machines would still be hard.
    • virgilp 5y ago
      Wait no, that can't be true. Finite state machines are only as powerful as regular expressions; you need pushdown automata even for programming languages, surely you can't model natural language with finite state machines? (maybe an infinite number of them, but I'm doubtful even on that - I'd have to review the theory to be sure though)
      • frostburg 5y ago
        It is true; keep in mind that real computers are strictly speaking FSMs, not Turing machines, due to having a finite "tape". It's just vastly more useful to think of them as such when programming.
      • skybrian 5y ago
        In practice, both people and machines can only handle a finite amount of nesting, so you could do it though it would be awkward to express.
      • robertlagrant 5y ago
        Perhaps the fact that some words can never follow other words would make it tractable.
  • ma2rten 5y ago
    Maybe, but that is not helpful in the sense that it doesn't guide research in how to get there.
  • jensgk 5y ago
    Yeah, I would even say a large amount of NAND gates should do it.
  • otabdeveloper4 5y ago
    You're assuming that intelligence is a computational process, but the sum total of what we know about intelligence says it probably isn't. (Unless you're making a more general reductionist statement that everything in the universe is a computational process - that kind of reductionism is understandable coming from people who work with computers for their job - but this is then a philosophical stance, not scientific, and frankly a very strange one.)
    • anyfoo 5y ago
      > but the sum total of what we know about intelligence says it probably isn't Source? I am not aware of any other known process in the universe that could not be simulated by a Turing machine.
      • otabdeveloper4 5y ago
        Assuming that the universe can be simulated by a Turing machine is a strong and weird claim that needs to be defended, not the other way around. We know that Turing machines are very limited things and that the computational processes they carry out also very limited in applicability. What's the evidence that the universe is more limited that a Turing machine? Just the fact that we can imagine things that can't be computed by a Turing machine should clue you in that it's probably otherwise.
        • anyfoo 5y ago
          > We know that Turing machines are very limited things and that the computational processes they carry out also very limited in applicability. Again, source? Do you know of anything that is able to perform a computation that a Turing machine cannot? > Just the fact that we can imagine things that can't be computed by a Turing machine should clue you in that it's probably otherwise. Like what? Uncomputable numbers like Chaitin's constant? We can "imagine" them by stating their definition, but we cannot compute them. Or do you have something else specific in mind?
          • 5y ago
  • tsimionescu 5y ago
    Technically that is a conjecture known as the Church-Turing thesis, which states that any computable function can be computed by a Turing machine. Here "computable function" uses the informal definition 'any function that a human can compute', not the formal definition of the recursive functions (which are proven to be solvable by a Turing machine).
• hervature 5y ago
  I guess things are slowing down at DeepMind. I have tremendous respect for David Silver and his work on AlphaZero and Richard Sutton as a pioneer in RL. But the cynic in me is that this paper is just a result of Goodhart's law with publishing count as a metric. Any proof of the type of emergent behaviors that they mention from RL with an actual RL experiment would go a long way. Showing an RL agent developing a language would be extremely interesting. It makes me think they tried to show these emergent behaviors but could not and thus ended up with a hypothesis.
  • maiodude 5y ago
    They just "solved" protein folding late last year. How can you say things are slowing down? Do you honestly expect life-changing discoveries every other week?
    • 0xB31B1B 5y ago
      Protein folding is a well modeled math problem. The alpha fold solution is extremely good at pruning (aka guessing) folding chain structure possibilities. I am Impressed and this is a difficult problem but this is extremely different from AGI as this is a well scoped easily modelable problem that is basically a chain of 26 inputs types of links of arbitrary length. I am not trying to take away that the protein folding is incredible but AGI is extremely different. AGI is literally having a model that can both do alpha fold and self driving cars, as well as the ability to generate novel models to solve new well scopes problems. RL can do 0 to 1, the 1 to n (generalizeability) is the extremely difficult part.
      • ackbar03 5y ago
        I may be really only speaking for myself here but I have very sincere doubts that anyone who has done any moderately serious work on "AI" and wrangled with the nitty gritty details of it all is really having any huge expectations for AGI. I mean if it comes during my lifetime, hurrah! But personally I'm not gonna sit around waiting for it or depend on it for anything. That being said, is the current AI tech as we have it useless? Of course not. Things like protein folding and alphago are still huge leaps forward in tech, it'd be kind of silly to treat AGI as the only thing worth achieving
• andyxor 5y ago
  good luck with that. DeepMind should sponsor B. F. Skinner award, to honor the father of their behaviorist theories of 'reward and punishment' as a sort of all-encompassing theory of everything related to cognition. At least now they are torturing GPUs and not some poor lab animals. on a serious note the only positive outcome of all this shameless PR is that the heavy investment in ML/RL might trickle down to actual science labs and fundamental neuroscience research which might move us forward towards understanding natural intelligence, a prerequisite for creating an artificial one.
  • eeegnu 5y ago
    > towards understanding natural intelligence, a prerequisite for creating an artificial one. I've thought about this before, and I'm not convinced it's really prerequisite. Naturally developed intelligence in my mind may actually be highly constrained and inefficient because it was limited to what was biologically feasible. i.e. There may be simpler ways of achieving comparable results. Natural intelligence does however have the benefit of being an actual working model, but deciphering the blackbox may be just as hard as developing a working theory from first principles.
    • andyxor 5y ago
      yes, it's a recurring thread, "do we really need to mimic the birds in order to build airplanes", etc. I think someone serious about AI should treat it not as engineering problem but as a science, like physics, which starts with model of nature, and experiment to prove or disprove the theory. Nature provides the constraints by which theory is developed, which radically limits the "search space" of theories. Otherwise it's a bit like throwing things on the wall and see what sticks, which is the primary method of current AI research.
      • Peritract 5y ago
        Mimicking birds wasn't necessary for flight. However, understanding them absolutely was; we didn't end up taking exactly the same route to the sky, but we absolutely learnt from birds on the way.
• ma2rten 5y ago
  I'm more in Yann LeCun's camp who called unsupervised learning the cake, supervised learning the icing and reinforcement learning the cherry on top of the cake.
  • hervature 5y ago
    My opinion is that view is very simplistic and unnecessarily offensive to a whole class of researchers. MuZero, developed by David Silver, uses a combination of RL, supervised learning, and unsupervised learning (state representation) coupled with a planning algorithm. It accomplished things far beyond anything unsupervised learning can ever accomplish.
    • Der_Einzige 5y ago
      Unsupervised learning is exactly the wrong way to approach chess or other games that MuZero solves. It's also worth noting that traditional alpha-beta pruning + heuristics are basically neck and neck with the very best of neural network based techniques. I'll trust stockfish over a alpha-zero or MuZero for awhile longer if I'm trying to win a computer chess competition ...
      • hervature 5y ago
        Sure, Stockfish just uses millions of years of evolution to build its heuristics and can't be transferred to any other game. The point remains, calling RL a cherry on the cake compared to unsupervised learning when they are completely orthogonal and not mutually exclusive techniques is simplistic and unnecessarily offensive.
        • unishark 5y ago
          So it's bad to be the cake? I assume he means it's the foundation one falls back on when the more specialized categories of methods are not applicable. You might not like my analogy either. I think of supervised and unsupervised learning as the majority of the genome of ML, while RL is that little Y chromosome sometimes tacked on to address a few high-profile tasks.
• throwaway879 5y ago
  If you send a message in a bottle it will eventually land ashore somewhere, maybe in a century, who knows, and who knows whether it will be relevant by then or not, or civilization may not even exist by then, but sure it's similarly plausible to get to AGI via RL.
  • visarga 5y ago
    > send a message in a bottle Just one - yes. But how about if you send millions of bottle messages?
    • ausbah 5y ago
      then you're in your way to a hit rock song!
    • throwaway879 5y ago
      >>how about if you send millions of bottle messages? assuming we can integrate all learnings from those bottles into a system that can classify any given situation and apply the learning in that domain. But to build a system that can classify any problem is where we're stuck at and RL can't get us there
• blueblisters 5y ago
  Current state-of-the-art in reinforcement learning can barely make a physical robot walk. In theory, with transfer learning, we will probably see better success over time but I'm looking forward to seeing results in practice. A 2018 article about the challenges of reinforcement learning: https://www.alexirpan.com/2018/02/14/rl-hard.html https://www.alexirpan.com/2018/02/14/rl-hard.html
• azinman2 5y ago
  Everything old is new again. As far as I can tell, they're not actually proposing how to achieve this. I can't access the article without a host institution it seems (is there another link?), so I only have the article to go by. RL has been the basis for all robots engaging with the world, and that engagement with the physical world modeled using RL has been promised to make robots that can act like a 2 year old for a long time (see Cynthia Breazeal's work, for example). Yet AFAIK, we haven't actually achieved this as we don't know how to efficiently model the problem to have learning rates that reach anywhere near what we're able to do with DNNs today. Perhaps someone who has access to the paper can say why this is a milestone? If Patricia Churchland suggests it is, then something new must be happening here.
  • hervature 5y ago
    I don't know who Patricia Churchland is, but they said that the paper was "very carefully and insightfully worked out." After having read the paper, I am very disappointed in the output. Nothing concrete was shown, just hypothesis and reads more like philosophy. That being said, I would say that the paper is carefully worked out and does provide insight if you haven't thought about RL before.
    • Ieghaehia9 5y ago
      If Patricia Churchland doesn't have a problem with the paper despite it being philosophical, then that's probably because she is a philosopher. An eliminative materialist, to be precise. Personally, from reading the abstract, I disagree with the hypothesis. There's a trick where anything (even say, a database lookup) looks like optimization as long as you contrive the objective function just right, but that's kind of uninformative.
      • LesZedCB 5y ago
        and our intelligence is quite good at confabulating optimization functions for abstract processes that are inert. pretty amazing, really.
  • leto_ii 5y ago
• harry8 5y ago
  Some Bozo who has heard all this many times before is suspicious of claims from places like Deep Mind who have a financial incentive to make them (keep funding) where there aren't working machines to back that claim up. Some Bozo has no credentials, no reputation, no track record of publications and barely supports the claim they're making with anything much. Some Bozo has no financial incentives or otherwise to opine either way. Some Bozo doesn't even work in the field at all. Bets: Some Bozo or Deep Mind turn out to be closer to being correct in the passing of some finite amount of time? 5 years? 10 Years? 25 Years?
  • rusk 5y ago
    Edit: sorry just realised you’re making the same point as me more or less. Putting yourself in third person. I’ll let my comment stand anyhow :) Screwing my face up, looking at this sideways … but it seems as though you’re saying that the Bozos of HN have nothing useful to contribute to this discussion based on … [rereads] … their lack of academic credentials in the area… you could say this about just about any HN post I’m just wondering why this one? Here’s a thing though … if the understanding of a technology is so nuanced … that Bozos can’t “get” it … is it really that mature? We had functioning computers for 50 years but it was only when the Bozos got their hands on it that things took off. Internet for 20. Cell phones for 10. How long are we dabbling with neural networks? 50 years or so? All I see in this most recent explosion in AI is a rapid jump in the availability of cores. Ala Malthus once that newly available “source of nutrition” has been used up we will see a rapid die off once more and it will be another 20 years once the Bozo intellect has caught up before we look at this topic en masse again. Dismiss the Bozos at your peril. You’re dependent on them for innovation and consumption. Your sincerely, a Bozo.
    • A-Train 5y ago
      The algorithms to train, initialize the networks, new architectures are far more important than the hardware advances. If people knew how to train NNs 50 years ago we would live in a different world.
      • chrisco255 5y ago
• dannyw 5y ago
  Alternate title: DeepMind fails to make progress on AGI, publishes thought piece instead.
  • visarga 5y ago
    And the entitlement we have is even higher than the difficulty of the task and the hard work people are putting in. Anyone here can say they did as much for RL?
• jjjdjjddddsfsd 5y ago
  Are the just reformulating the principles of evolution in digital terms, and essentially not providing any new insights at all? Yes, intelligence has been created by evolution. That doesn't imply that any system that is subject to evolutionary forces will lead to the creation of intelligence (and not within a reasonable timeframe, either). The challenge is to create a system that is capable of evolving intelligence. Afaik some biologists even think that the evolution of intelligence was rather unlikely and would not necessarily happen again under the same circumstances as on earth.
  • visarga 5y ago
    Evolutionary algorithms are tricky, just like deep learning. It's not "just reformulating the principles of evolution in digital terms, and essentially not providing any new insights".
  • webmaven 5y ago
    > Afaik some biologists even think that the evolution of intelligence was rather unlikely and would not necessarily happen again under the same circumstances as on earth. Hmm. Can you provide a pointer to those biologists? AFAIK, high intelligence has arisen more than once on Earth (Hominoids, Cetaceans, Octopuses), so I'm somewhat skeptical of that claim, but perhaps they're construing intelligence more narrowly (ie. only Homo Sapiens qualifies).
    • jjjdjjddddsfsd 5y ago
      Jared Diamond talks about it in his books (don't remember which ones specifically). OK, granted, he is not officially a biologist, I guess, but at least a prominent writer on Evolution Theory.
    • rubatuga 5y ago
      Well that has a prior on life even existing in the first place
      • webmaven 5y ago
• hyfgfh 5y ago
  “Talk is cheap. Show me the code.”
• Sparkyte 5y ago
  TLDR, general AI means simplistic AI.
• adipandas 5y ago
  This article is interesting, I even skimmed through their paper. But I think still the question remains: How to find the unified reward function? Or in other words, how to find answer to life? [It cannot be 42].
  • goldenkey 5y ago
    Intelligence would be produced in any Turing complete automata. But the universe has a frame rate of 10^34 (based on Planck constant.) We don't really have the tech to just run "evolution" of a universe or of even a psuedo biological substrate.
    • drdeca 5y ago
      This seems far from clear. Just because a system is capable of turing complete computation does not imply that a generic state of the system will typically eventually produce intelligence or even something which is sophisticated in some sense. As a trivial example, consider a variation of Conway's game of life which, in addition to black and white cells, also has green cells, where any cell next to one or more green cells will be a green cell in the next time step. A generic state in such a variation will have at least one green cell, and therefore all parts of it will eventually be green, and so no useful long running computation will be done, certainly none which takes where the green cells are into account. But, such a system would still be turing complete, because one could start in a state in which there are no green cells, and in those states you just have Conway's game of life. That trivial example works as an existence proof, but even for less extreme cases it isn't clear. Consider ordinary conway's game of life. To paraphrase a question from Alex Flint on Alignment Forum (https://www.alignmentforum.org/posts/3SG4WbNPoP8fsuZgs/agency-in-conway-s-game-of-life https://www.alignmentforum.org/posts/3SG4WbNPoP8fsuZgs/agenc... ) Suppose we have some 10^50 by 10^50 square where an agent is supposed to be implemented, and this 10^50 by 10^50 square is at the top left corner of a, say, 10^100 by 10^100 square, where the rest of the square is initialized randomly, is it even possible for the agent to be such that it has a high chance of successfully influencing the large scale state of the rest of the 10^100 by 10^100 region in the way that is desired? It isn't clear. It isn't clear that a structure can withstand the interactions with a surrounding chaotic region. Perhaps some systems are such that they do allow Turing-complete computation, and are such that typical states result in complex behavior, but are also such that all really structured behavior is always very "fragile", and can only continue in a structured way if what interacts with it is in a small set of possible interactions. To be capable of Turing complete computation, is not, I think, sufficient for "life" (a self-maintaining thing) to arise from typical/generic states, even when under the assumption that typical/generic states lead to continually complex behavior (to exclude the spreading green cells case) Also, I don't think we can confidently say that the Plank time is "the universal frame rate". Better to refer to Bremermann's limit and the Margolus–Levitin theorem , though these bounds depend on the amount of energy available. (10^33 operations per second per joule, where the energy is the average energy of the system doing the computation)
• webmaven 5y ago
  Saying RL is sufficient to (eventually) achieve AGI is a bit misleading. One might similarly state that biological evolution is sufficient to (eventually) achieve biological general intelligence. Both statements are probably true, but the parenthetical (eventually) is doing an awful lot of heavy lifting.
  • neltnerb 5y ago
    Yes, it's easy to be convinced on either side, the arguments write themselves. Yes, eventually a learning system might learn enough to be indistinguishable from intelligence. Or this might be entirely the wrong path and detracting from genuine new innovations in how we think about AI. We won't be able to tell whether it's AGI or just good enough at trained tasks to trick us.
    • exporectomy 5y ago
      It can prove its intelligence by making testable predictions of the future better than us. As for whether it's "real" AGI or just acts like it, doesn't really matter. I think the Chinese room problem has been agreed on as not a problem, hasn't it?
      • neltnerb 5y ago
        My last sentence was a statement of that problem, not a question.
      • chriswarbo 5y ago
        > As for whether it's "real" AGI or just acts like it, doesn't really matter. Absolutely. The term "AGI" came about specifically to avoid existing philosophical arguments about "strong AI", "real AI", "synthetic intelligence", etc. Those wanting to discuss "true intelligence", etc. should use those other terms, or define new ones, rather than misuse the term AGI. AGI requires nothing more (or less!) than a widely-applicable optimisation algorithm. For example, it's easy to argue that a paperclip maximiser isn't "truly intelligent", but that won't stop it smelting your haemoglobin into more paperclips!
• Animats 5y ago
  "A sufficiently powerful and general reinforcement learning agent may ultimately give rise to intelligence and its associated abilities. ... We do not offer any theoretical guarantee on the sample efficiency of reinforcement learning agents." OK. This basically says "evolution works". But how fast? Biology took tens of millions of years to boot up. An related question is how much compute power does evolution, viewed as a reinforcement learning system. have? That's probably something biologists have thought about. Anyone know? Evolution is not a very fast or efficient hill-climbing system, but there are a large number of parallel units. It's not a philosophical question; it's a measurable one. We can watch viruses evolve. We can watch bacteria evolve. Data can be obtained. Two questions I pose occasionally are "how do we do common sense, defined as not screwing up in the next 30 seconds", and "why does robotic manipulation in unstructured situations still suck after 50 years". A good question to ask today is why reinforcement learning does so badly on those two problems. In both cases, you can define an objective function, but it may not be well suited to hill climbing.
  • londons_explore 5y ago
    In some cases biological 'genetic algorithm' hill climbing can be remarkably ineffective. For example, the classic "design a car that can drive over this terrain" problem, even after a billion generations (~ the same number as life on earth), shows no substantial performance improvement. That makes me suspect something is missing from our biological genetics model.
    • wnkrshm 5y ago
      energy supply and other constraints (material, robustness ...) are a good explanation though - an organism can't grow out of aluminum or steel
      • EE84M3i 5y ago
        I suspect they're talking about this (sort of) simulation https://rednuht.org/genetic_cars_2/ https://rednuht.org/genetic_cars_2/
    • 5y ago
• nathias 5y ago
  I believe the goal of 'general' AI will have to be modified once we know more, what appears as general is probably a set of discrete faculties that play well together.
• Barrin92 5y ago
  Sorry but where is actual scientific content in that paper? I'm concerned with the state of AI. saying that "reinforcement is all you need", when reinforcement learning is defined as abstract as "agent does something, adapts to environment and rewards, then does another thing" is borderline tautological. The actual scientific question is, what are the mechanisms that make agents work, what are the fundamental modules within intelligent systems, is there a distinction between digital and biochemical systems, what costs are there in terms of resources and energy to get to a certain level of intelligence, and so on. Real questions with specific answers. For all the advances coming from just upping the amount of data and GPU hours, there is so little progress on trying to have a model of the structures that underpin intelligence.
  • zzzzzzzza 5y ago
    i think part of what they are saying is that your approach is wrong, (e.g. looking for then copying submodules within intelligence won't generalize), trying to answer specific questions won't generalize, but if you train a network with the right potentially hacky series of rewards/rich enough environment you could get a much more general intelligence a new kind of science
• littlestymaar 5y ago
  Being “enough” isn't practical though and whether it's sufficiently efficient is still an open question. We already know that evolutionary trial and error is “enough” to create flying animals, yet with a little bit of domain knowledge we've been able to build planes way faster than nature has built flying insects.
• twhitmore 5y ago
  While I am agnostic as to the advantages of RL over simpler ML, some basic structural concerns seem to apply here. The potential concern is that if RL works, the very best factors for learning are likely to be potentially dangerous. The best environment for learning is the richest -- ie. unrestricted access to be the entire Internet/ or the world via the Internet. The best reward function is quite likely to be reproduction. If the researcher allows moderate action but attempts to limit reward functions, at a certain level they may find the AI has found a better reward. If the claimed advantages to learning of richer environments & richer action capability are even somewhat true, researchers are likely to be strongly incentivized to pursue such risky structures.
• sam-2727 5y ago
  By far not an expert in this area, but if (when?) we successfully generate true intelligence, I suspect it will be through some sort of "ensemble" model, where multiple agents are trained in parallel and interact with each other. Intelligence as we know it hasn't just resulted from an evolution of one agent in response to a cost function, but rather through the complex interactions of agents (humans and organisms in general) over time. I feel like the underlying journal article (https://www.sciencedirect.com/science/article/pii/S0004370221000862 https://www.sciencedirect.com/science/article/pii/S000437022...) is missing discussion of this.
• carrolldunham 5y ago
  Isn't this tautological or trivial because any proposed alternative requirement than reward maximisation can be reframed as maximising a different reward?
• hestefisk 5y ago
  For those of you ‘generalised’ AI sceptics like me, I can highly recommend reading Søren Brier’s book on Cybersemiotics (“why information is not enough”). It’s a comprehensive reader into the physicalistic, reductionist field of AI and all of its shortcomings. General AI implies the ability to abduct (not just deduction and induction), which I highly doubt will ever be possible.
  • funklute 5y ago
    > which I highly doubt will ever be possible What prevents an AI from performing abduction? While I've never thought about it before, it intuitively seems like a pretty straightforward thing to implement....
  • reedf1 5y ago
    How does he resolve that humans achieve the ability to "abduct"? I just don't buy that there is something ineffable about humans - and even if there is, why can't we just plug that process into a computer?
  • cromwellian 5y ago
    "never" (I know you didn't say that, but it's implied), is a pretty strong claim. Like with Searle's Chinese Room, I find these kinds of impossibility arguments fairly weak and full of semantic problems. Whenever people argue "consciousness", or "general AI", or "meaning", it's always an exercise in moving the goal posts. When I read abstracts like this: "Cybersemiotics constructs a non-reductionist framework in order to integrate third person knowledge from the exact sciences and the life sciences with first person knowledge described as the qualities of feeling in humanities and second person intersubjective knowledge of the partly linguistic communicative interactions, on which the social and cultural aspects of reality are based. The modern view of the universe as made through evolution in irreversible time, forces us to view man as a product of evolution and therefore an observer from inside the universe. This changes the way we conceptualize the problem and the role of consciousness in nature and culture. The theory of evolution forces us to conceive the natural and social sciences as well as the humanities together in one theoretical framework of unrestricted or absolute naturalism, where consciousness as well as culture is part of nature. But the theories of the phenomenological life world and the hermeneutics of the meaning of communication seem to defy classical scientific explanations. " My confidence isn't increased that this book has anything interesting to say. It sounds like the kind of post-modernist verbiage coming out of a lot of humanities departments. I mean seriously, why anyone in a STEM field actually write a sentence like "modern view of the universe as made through evolution in irreversible time, forces us to view man as a product of evolution and therefore an observer from inside the universe." Like what is "irreversible time" adding there? It's kind of redundant, and the awe-inspiring implication: man is a product of evolution, and therefore an observer in the universe? Earth shattering. And all phenomena, from fundamental life sciences to culture, are part of nature? No one's ever considered that before. My guess is, if you asked a STEM major to write this abstract, they could sum it up with 70% less words. They'd also drop implicitly obvious claims that no one disputes. I'm betting that this is book is just a less philosophically rigorous version of Searle's argument, given the implication that emergent phenomena "seem to defy classical scientific explanations".
• kakadzhun 5y ago
  Even if Reinforcement Learning is 'enough', it will be held back by whatever methods used to implement it (e.g. Deep Neural Networks). As the algorithms get more advanced, from some point onwards, to build the general AI, you first need a general AI to tell you the correct hyperparameters so that the pile of methods would work well in tandem. Last but not least, you will be bound by your inability to accurately communicate exactly the behaviour you want out of the AI because you are incapable of writing down a mathematical function that would induce the behaviour in the learner. And then you wonder why the general AI decided to pull the plug on your grandma and try to use that resource for something else instead.
• rich_sasha 5y ago
  Basically, any problem with a solution fits into RL: reward of 1 if you are AGI and 0 otherwise. Go learn. This setting on its own is meaningless! The “how” of the RL agent is not even 99% of the problem, it is all of it. Given our understanding of both DL and neuroscience, it is not even clear to me that we can say with confidence that Neural Networks are a sufficiently expressive architecture to cover an AGI. The human brain is a deep net, sort of, but there is also plenty going on in our brains that we don’t understand. It could be that the magic sprinkle is orthogonal to DL and we just don’t know about it yet.
  • svantana 5y ago
    > The “how” of the RL agent is not even 99% of the problem, it is all of it I'm not sure that's true anymore - pretty much any objective devised is being solved by ML solutions within months (with some exceptions such as Chollet's ARC, maybe Winogrande). But those same models will perform poorly on other unseen tasks, because ML takes shortcuts if it can. We used to have unsolved tasks for decades, such as Go. It's now comparably hard (if not harder) to create a good objective measure of intelligence than to reach human parity on said measure.
    • rich_sasha 5y ago
      Maybe I’m misinterpreting your comment, but what about self-driving cars? Hugely researched and funded area, not without success, but even the best self-driving system is so inferior to a human driver. I’m not dissing self-driving car research, just not sure we’re anywhere close to parity, and the problem is fairly well defined.
      • svantana 5y ago
        Right, I was talking about mathematically defined objectives and inputs (algorithms & data, in practice). With self-driving, arguably the main stumbling block is that the objective is not well-defined and the inputs are potentially wrong (faulty sensor data).
  • cynusx 5y ago
• AndrewOMartin 5y ago
  Hydrogen and time "enough" to reach general AI. https://xkcd.com/1123/ https://xkcd.com/1123/
• im3w1l 5y ago
  So regarding objective function, one idea I just had is this: Teach them warfare. To quote a cliche: "we live in a society". As humans we are embedded in a social environment which has a few important features: We cooperate, we compete and we die. These three pillars are the basis of our culture (a concept we should apply to AI btw). Because of competition we are forced to learn everything there is to learn (general intelligence), to get a leg up. Because of cooperation and death we need to continuously transmit and share knowledge with our friends and the next generations. Ever changing alliances means we need to get good at both deception and detecting it. For this reason I think warfare is ideal for reaching general AI.
• seaorg 5y ago
  The layman and the AI expert have both written off the possibility of creating AGI for the entirety of this latest AI spring. In the past decade I have basically never encountered anyone who thought that AGI was going to happen in our lifetimes or even anyone who believed that it would be a problem if it did. One time I discussed AGI with a good friend. And I gently pressured him to play through the scenario of the advent of AGI. And he made a guilty confession to me. I could tell he was embarrassed to share his opinion. He said “can’t we just unplug it?” This is a microcosm of the entire issue. It’s something a child might say. For global warming, can’t we just turn on the air conditioning? No, we can’t just unplug it. Here is how you can understand what is happening. An AI model is just a program. A program that is written by another program. Consider a list of programs that contains every possible program. It is infinite. Make it finite by limiting the size of the programs based on some practical consideration. It’s still a very long list. What is in this list? Pong. StyleGAN. AGI? Almost certainly. How small might AGI be? It probably appears more than once on the list — how many are there? But let’s ask the real question here and boil this down. How many items on the list are something we will regret having discovered? A great, great many. Every day, people are exploring the list. They are using primitive methods to sift through all these programs and find the ones that have interesting behavior. The process of program discovery is automated with things called “training algorithms.” As our computers get faster and capital allocation grows larger we mine this list more and more quickly. The problem is that we keep finding things that surprise us. And that is the core and substance of the entire issue. We have demonstrated to ourselves over and over again that our own intuition about the contents of this list is completely wrong. Despite the mind-boggling level of mental gymnastics performed in the wake of GTP-3, enough to constitute a three ring circus, the fact of the matter is that GTP-3 and soon to be 4 is not supposed to exist according to every single AI expert. Compute is only getting cheaper and capital is only getting bigger and the list is only getting shorter and it’s not going to stop surprising everyone including the “experts.” Just because we don’t know how to make something we will truly regret doesn’t mean we aren’t capable of creating it.
  • helen___keller 5y ago
    > He said “can’t we just unplug it?” This is a microcosm of the entire issue. It’s something a child might say. For global warming, can’t we just turn on the air conditioning? No, we can’t just unplug it. If it were running as a sandboxed application inside some kind of runtime environment with only access to print output text and read input text, as you might expect a GPT-n program to run, certainly you could unplug it. Humanity could be considered God's AGI, but if God doesn't give us the APIs to interact with heaven, it follows that we can't kill God. Of course, humans are much more greedy, so I'm sure our creations will have full access to any APIs needed to run our factories, advertise products on the internet, and maybe direct our military weapons too :) Oops! > Despite the mind-boggling level of mental gymnastics performed in the wake of GTP-3, enough to constitute a three ring circus, the fact of the matter is that GTP-3 and soon to be 4 is not supposed to exist according to every single AI expert Could you elaborate? What mental gymnastics, and it is not supposed to exist according to who exactly?
• orwin 5y ago
  Reinforcement learning without auto-growing inference engine embedded? Yeah, no chance its enough.
• jgalt212 5y ago
  It's not far from the truth to say that Deep Mind is selling snake oil (AGI) and delivering vegetable oil (SOTA pattern recognition software).
• robomartin 5y ago
  Yeah. No. No way. My son and I were discussing the state of AI/robotics yesterday, as we walked a beautiful trail in the Sequoia National Forest. What prompted the discussion was a simple question: What would it take to build a robot capable of navigating these trails as we do? This would be a robot able to do this in a manner indistinguishable from, say, a ten year old human. No GPS, maps, compass, pre-mapping, lidar, ultrasonic sensors, etc. Just vision, hearing and touch/force sensing at the “skin” and articulations. What do you know? Two things: You are located at the start of the correct trail and there’s a waterfall at the end. Our conclusion was equally short and simple: Today, it is hopelessly impossible to match what a ten year old kid could do on that trail. Maybe in ten years. Maybe. AI today can’t do what a ten year old human, or the young bears we so along the trail, can do instantly and without thinking: Understand. We just don’t know how to approach and encode understanding yet.
• mirker 5y ago
  Sufficiently large boolean satisfiability also enough to reach general AI. I kid, but these sorts of papers are theoretical position papers and do not account practical considerations e.g., for how many suns of energy must be expended.
• scythmic_waves 5y ago
  /r/MachineLearning discussion: https://www.reddit.com/r/MachineLearning/comments/nplhy3/r_reward_is_enough_david_silver_richard_sutton/ https://www.reddit.com/r/MachineLearning/comments/nplhy3/r_r... I'm with most of the comments there. This paper is ridiculously hand-wavey.
  • JamilD 5y ago
    Many of DeepMind's opinion style papers are like this. Another example of the "handwavy" DeepMind paper: https://arxiv.org/pdf/2102.03406.pdf https://arxiv.org/pdf/2102.03406.pdf It's also worth it to note as well that this isn't a homogenous organization, many DeepMind employees have different opinions on issues like this and an individual paper isn't representative of the entire organization.
    • scythmic_waves 5y ago
      > It's also worth it to note as well that this isn't a homogenous organization Please don't consider my critique of this paper as an indictment of DeepMind as a whole! > Many of DeepMind's opinion style papers are like this. That's good to know. I have not read many of their opinion papers, and I'll admit I didn't have the context of it being an "opinion" paper. That said, I don't agree with the opinion. The paper didn't really engage with the concept of AGI in a way that I found satisfying. The conclusion may very well be correct, but this paper wasn't enough to convince me. Slightly OT: My views were reinforced when I saw the paper was praised by Patricia Churchland. I don't find her take on consciousness a satisfying one, though I find the general direction of her work interesting. See here for another example: https://www.reddit.com/r/philosophy/comments/nvtgwr/grand_theories_of_consciousness_will_not_help_us/ https://www.reddit.com/r/philosophy/comments/nvtgwr/grand_th...
• 21eleven 5y ago
  RL can provide amazing results (AlphaGo, AlphaStar (Starcraft 2 agent), etc) but it requires a well modeled world to work with. Games like Go and Starcraft are well modeled worlds. If you want something akin to AGI to operate in the "real world" you will need a high quality data model of the real world for the RL system to work off of.
  • atombum 5y ago
    Agreed, and that brings up a very interesting discussion of prejudice in modeling the world. Everyone views the world differently and it seems to me of vital importance that any attempts to model the real world for RL are as unbiased as possible. Or more realistically, incorporate as many different biases as possible.
• EGreg 5y ago
  AlphaGo can play games against itself. You can have GANs, MCTS and more. General AI requires a feedback mechanism from the real world. Unless you have an accurate model of it in a computer, you can’t just test whether a joke will be funny without waiting for humans to laugh. You can’t check whether a tailored diet or workout regimen or gene therapy will have good results without humans trying them. So you’ve reduced your AI problem to a harder problem: modeling the world and all of its complexity in a computer, and somehow being able to run simulations faster than the stuff that happens in the actual real world
• btilly 5y ago
  My first response is that B. F. Skinner would be proud. I'm going to predict that reinforcement learning will be important but not sufficient. Just as behaviorism is important but not sufficient for psychology.
• fooker 5y ago
  https://xkcd.com/1123/ https://xkcd.com/1123/
• deleted 5y ago
  [deleted]
• sklearncowboy 5y ago
  What proof is there that I am generally intelligent in the same context we want computers to be in order for AGI to be true? Imagine how much farther the field would be if we stopped wasting resources on this fantasy land nonsense. All intelligence is specialized intelligence.
  • jawarner 5y ago
    Yes, human intelligence is the benchmark for the definition of general intelligence. Certainly we can go further.
• bitL 5y ago
  RL is still pretty dumb and for that extremely computationally demanding. Markov chains are like the most trivial stochastic models and I can't believe that's all it takes to get AGI. I also often wonder whether the "Deep" part of DRL is not spending most of its weights to compensate for the basic nature of Markov chains.
• The_rationalist 5y ago
  Cringe
• woeirua 5y ago
  If RL is enough, then there is no physically realizable way to actually train a RL based GAI in the near future. RL based learning requires evaluating the outcome of millions or billions of scenarios over time in order to optimize the network. Given that requirement you'd have to either find a way to accurately model the world and all of those interactions in silicon, or you'd have to build millions of robots that can report back the results of billions of interactions each day. It's not impossible to do that, and maybe it would even be likely that we would eventually accomplish that but the cost would make it prohibitive for anyone but a nation to even attempt today. It's almost certainly outside the realm of what is possible in the near future. Maybe when robotics has progressed enough that robots are capable of interacting with the world with basic AI will we see the rise of something like a GAI.
• deleted 5y ago
  [deleted]
• LesZedCB 5y ago
  the paper postulates that "reward is enough" but what is the alternative? rewards plus what? the "cybernetic feedback loop" aka reinforcement learning has kinda always been the idea, no? i believe that architecture is important. yes, at one level the human is a rewards machine, but the neocortex produces a lot of "self-reflection" that doesn't really generate reward in itself, at least i don't know if it does. maybe meditations create a general dopamine spike?
  • exo-pla-net 5y ago
    One needs to actually read the paper to understand what it's postulating; the paper's title is not enough. There is a temptation to view our capabilities as a complex set of interacting modules. Under that view, each module would need to be separately developed in order to artificially recreate our abilities. The paper makes the case that we need only a single carrot. Per the paper: "For example, the ability of social intelligence has often been framed as the Nash equilibrium of a multi-agent system; the ability of language by a combination of goals such as parsing, part-of-speech tagging, lexical analysis, and sentiment analysis; and the ability of perception by object segmentation and recognition. In this paper, we consider an alternative hypothesis: that the generic objective of maximising reward is enough to drive behaviour that exhibits most if not all abilities that are studied in natural and artificial intelligence."
• mensetmanusman 5y ago
  Does anyone know if deepmind is working on driverless technology? It seems like one of the largest values they could bring to society would be to solve driverless technology. Currently their best partner would be Tesla due to the amount of data Tesla has, but I doubt Google would allow that collaboration…
  • X6S1x6Okd1st 5y ago
    https://www.businessinsider.com/deepmind-is-teaching-googles-self-driving-cars-to-get-smarter-2019-7 https://www.businessinsider.com/deepmind-is-teaching-googles...
  • eatmyshorts 5y ago
    I'm curious why you feel their best partner would be Tesla instead of Alphabet's own Waymo?
    • mensetmanusman 5y ago
      Because of how much data Tesla’s fleet is generating every day (gathering while the user drives).
• bgroat 5y ago
  I'm not a neuroscientist, an AI specialist, or a hardware engineer. But as an enthusiast of all three I really think that AGI is a hardware problem, not a software problem. Reinforcement learning on a massive corpus of data is how we train all biological intelligence. The crazy thing is that in humans we manage to do it on ~3 watts an hour. I think we have the software cracked, my gut thinks silicon just isn't the right material
  • pkAbstract 5y ago
    You may be right, but it's also commonly believed in these communities that hardware is the part that's already been solved. Computer hardware already vastly outstrips human capacity in many domains. To me, it seems more likely that we're missing something/some things on the software side. AGI could probably run on present day hardware or even older.
  • 6gvONxR4sf7o 5y ago
    Silicon is likely fine as a material. GPU cost per operation is still dropping insanely quickly. A lot of really hard ML problems are just making big things feasible, or sampling big things enough to get decently precise estimates. With 10x the GPU power and memory, a lot of this gets easy. With 100x, some hard things get trivial. At the end of the day, GPUs and TPUs drive AI research more than anything else as models grow massively.
• rexreed 5y ago
  It is not enough. Machine learning only gets you so far, even the best general-purpose algorithms. Deriving reasoning and common sense demands something more than learning. I agree with the other commenter here that the smartest systems are less intelligent than the common housefly. A brain the size of a pinpoint can navigate, eat, reproduce, and live a full life without big data or internet.
  • ggggtez 5y ago
    On the one hand; award winning ai specialists. On the other: an anonymous internet commenter. I'm not going to take bets on who is right, but simply saying "nuh uh" is not exactly breaking ground.
    • deleted 5y ago
      [deleted]
  • for_i_in_range 5y ago
    I concur. There is “Reality”, and then there is human’s conception of “Reality”. (See also: “Theory of Forms”). AI and Machine Learning are housed in human’s conception of Reality. Not Reality itself. We’re innovating in a sandbox, but we’re getting better and learning more to the point of getting so good, we may one day get out...
    • rexreed 5y ago
      There are so many people that conflate AI and machine learning and believe they are one and the same thing. They are not the same thing. Machine learning is a proper subset of AI. Even the best machine learning doesn't embody all of AI, because AI is more than just machine learning. Anyone who has spent more than a few years doing AI research and not just reading popular science blogs knows that AI is not the same thing as machine learning. Yet, here we are, saying that achieving one form of machine learning to its fullest extent will be "enough" to meet the challenge of AGI.
      • 5y ago
• abeppu 5y ago
  A lot of comments in this thread are talking about how unsatisfactory this paper is, because of course rewards are enough for _some_ agent, but this paper doesn't venture to say anything about that agent (e.g. what differentiates humans from squirrels even though both are trying to eat and reproduce?). But I think though they talk about rewards incessantly, the interesting angle is the importance of a complex environment in which the agent learns to maximize a reward: > we suggest the emergence of intelligence may be quite robust to the nature of the reward signal. This is because environments such as the natural world are so complex that intelligence and its associated abilities may be demanded by even a seemingly innocuous reward signal. On the one hand, this does echo some old work on situated cognition, which one might actually believe. But perhaps politically, if the claim behind the claim is that we can only develop powerful AGI which understands how to interact with our world by developing agents that learn with unfettered access to the world, then perhaps this will be the beginning of a strong push for tolerating spastic ineffective robots in our physical environments, and letting error-prone agents have vast access to our virtual environments. We'll be asked to put up with their mistakes because that's supposedly the cost of progress; limiting their environment would limit their cognitive potential. > For example, consider a signal that provides +1 reward to the agent each time a round-shaped pebble is collected. In order to maximise this reward signal effectively, an agent may need to classify pebbles, to manipulate pebbles, to navigate to pebble beaches, to store pebbles, to understand waves and tides and their effect on pebble distribution, to persuade people to help collect pebbles, to use tools and vehicles to collect greater quantities, to quarry and shape new pebbles, to discover and build new technologies for collecting pebbles, or to build a corporation that collects pebbles. Aren't you comforted they chose round pebbles instead of paperclips? And though their example is meant to illustrate that the reward function doesn't matter, you'll notice there's no negative reward term for e.g. smashing a retaining wall to dig for pebbles in the rubble, or dredging a beach where a protected bird species nests, etc. "Allowing the agent to fully explore the complex environment is the only way it will learn complex representations and actions!"
• ur-whale 5y ago
  >DeepMind says reinforcement learning is ‘enough’ to reach general AI When a company like DM makes such statements, you have to take into account the fact that they've essentially bet the farm and the neighbor's on RL. As such, the statement isn't really carrying much weight.
  • rsj_hn 5y ago
    What would have carried weight would be if this is something they actually achieved and then said "this is how we did it.". Making predictions of how they will succeed in the future doesn't really carry much weight, particularly in AI
• KiLo_H4CK 5y ago
  If these machines can don this, Then we are really in trouble.
• cowanon22 5y ago
  The AI field has a history of over confidence; just look up Marvin Minsky's predications in the 1950's and 1960's. This had been repeated throughout AI history - a new approach is found, it has some promising initial results, and then progress gets stuck and the approach is basically halted for decades. Machine learning and neural networks have prospered due to massive increases in computing speed and data, but I suspect it will also be a dead end in general AI. There will be some niche wins, some quite impressive, but the machine won't have the properties of an intelligent child or even a rat. There seems to be a big difference in how actual neurons work - I think neural networks are misnamed. The brain is not a big matrix, biology does not appear to reinforce behavior using gradient operations. I think increased research into neuroscience and biomimicry is the most likely approach for general AI, but I think we are still far away and the current hype will just be one more dead end.
  • nopeNopeNooope 5y ago
    There's realtime video processing that happens on smartphones to turn influencers into anime doll creations, seamlessly! Most people DON'T EVEN KNOW IT IS HAPPENING! The backgrounds on millions of peoples video conferences are shifted to something fanciful every hour. There's trillions of dollars of value created by automated software compositing images taken by capture cars, and turning that into points of interest and regular (automatic!) address updates. Ad engines are so incredibly effective, that inside a hundred milliseconds it can hold an auction and target you with things so perfectly tailored to what it knows about you that people are ABSOLUTELY convinced that Facebook and Google are listening in on them. And you think this is an over hyped dead end? Who cares how neural networks really work? This is good enough to get things done today, and likely well into the near future. I'll also point out that the lags in AI progress generally have more to do with the funding situation than the actual progress of the applications. It turns out when you kill the fountain of money that was Bell Labs, you have to wait until the fountain of money is reborn as Google. This isn't really a major difference from what we started with in the 50s, we're still using most of the same concepts, they'd almost certainly be instantly recognizable to folks if you chalk boarded it out for them. For you to say we hit a dead end back then when we're doing effectively the same thing, but with a much larger budget and talent pool is disingenuous.
• DonnyV 5y ago
  Survival at all costs will evolve by only basing general AI on only rewards. The system will start saying and doing anything it can to stay turned on. Not sure only basing general AI only on a reward system would be a good idea. Like the article says. There is no mention of the ability to empathies or give ones self up for a child or loved one.