13 ms·

Why traditional reinforcement learning will probably not yield AGI [pdf]

82 points by xamuel 6y ago

xamuel 6y ago
Author here. One particularly topical observation (topical because HN has recently featured discussions about ways to merge statistical and symbolic approaches to AI), from Section 4.2.3: certain cutting-edge number systems such as Conway's "surreal numbers" are so sophisticated that they require lots of symbolic logic just to do basic operations with. Of course this makes these number systems hard to work with, but they do not suffer the flaws pointed out in the paper, which the easier-to-work-with real numbers suffer. Since surreal numbers inherently require symbolic logical methods to work with, it follows that any sort of statistics-based RL agent for environments with surreal number rewards, would automatically combine symbolic logic and statistics.
- nx2059 6y ago
  The article was pretty over my head, but does your argument hold to the various augmented neural net systems such as Neural Turing Machines, or Differentiable Neural Computers?
  - xamuel 6y ago
    The argument isn't so much about the type of agent (which I think is what Neural Turing Machines etc. are about), it's about the type of environment. In traditional Reinforcement Learning, environments give real-number-valued rewards (or even rational-number-valued rewards which is even more constrained). Presumably this was a decision that was made with hardly a second thought because real numbers are most familiar to people... but such "appeals to familiarity" are totally irrelevant for such an alien field as AGI :) The point of the paper is that a genuine AGI should be able to comprehend environments that involve rewards with a more sophisticated structure than can be accurately represented using real numbers.
    - ssivark 6y ago
      IIUC, the claim is that the very idea of a (real valued) “objective function” to be “optimized” is broken?
      - xamuel 6y ago
        Broken in the sense that it's not flexible enough to apply to all conceivable environments a genuine AGI could navigate, without misleading that AGI. But I should stress that real number objective functions are probably fine for many specific interesting environments, I'm not trying to say that real number objective functions are useless. Just that they aren't flexible enough to cover all environments :)
        ssivark 6y ago
        Fair enough. It would be interesting/instructive to construct (relatively simple) examples where we can see that they’re broken :-)
        xamuel 6y ago
        Easy to come up with examples using exotic money-related constructions. Suppose there's something called a "superdollar". If you have a superdollar, you can use it to create an arbitrary number of dollars for yourself, any time you want, which you can trade for goods and services. If you want, you can also trade the superdollar itself. Now picture an environment with two buttons, one of which always rewards you one dollar, and the other of which always rewards you one superdollar. Shoe-horning this environment into traditional RL, you'd have to assign the superdollar button some finite reward, say a million. But then you would mislead the traditional-RL-agent into thinking a million dollars was as good as one superdollar, which clearly is not true.
    - IshKebab 6y ago
      But isn't the sophisticated structure what leads to the real number. If you change the rewards to something more complex surely you still have to pick between actions and at some point you'll have to evaluate which one is "better" and I can't see why you couldn't use real numbers to represent utility. I mean, humans are general intelligences, and you can translate pretty much any human reward into money, which is a real number. The paper is super long though. Maybe someone can give a TL;DR that makes sense.
- TTPrograms 6y ago
  I think you are hiding a key assumption - that an AGI must be capable of performing optimally on a task with Archimedean measure. This is a very strong assumption - performing eps-close to optimal may certainly fall in the definition of AGI, and this might be achieved by a non-Archimedean approximation. Defining AGI as performing optimally on any set of tasks is problematic from a computational theory perspective in general - even with real reward signals. Furthermore, infinite rewards are not compatible with human behavior. Humans never optimize for a single event at infinite expense w.r.t. other goals.
  - Animats 6y ago
    That an AGI must be capable of performing optimally on a task with Archimedean measure. If that's correct, this may be just one of those many problems where the optimal solution is far harder than a near-optimal solution. Examples include linear programming and the traveling salesman problem, where a true optimum is NP-hard to find, but you can get very close with far less work.
  - xamuel 6y ago
    Hi, thanks for looking at my paper. I do not assume that an AGI must be capable of performing optimally on all tasks in general--indeed, that's quite impossible. When measuring the performance of RL agents, one must come up with some way of aggregating performance across many environments, but that's beside the point of this paper. The point of this paper is that if you're forced to use real numbers as rewards, you can't even communicate all environments to the agent without misleading the agent. Whether the agent could perform well or poorly in the environments once communicated, is beside the point.
- bra-ket 6y ago
  do your claims apply in complex-valued RL context?
  - xamuel 6y ago
    I'm surprised to see that that's actually a thing people write about. I'm afraid I can't answer your question right now, as I have no idea why anyone would do that (as opposed to, say, vector-valued rewards---i.e., why is the ability to complex-multiply two complex rewards relevant, as opposed to merely comparing separate components of rewards from R^2). I'll have to read up on the subject :)
- gwern 6y ago
  "Assuming AGI agents are Turing computable, no individual AGI can possibly comprehend codes for all computable ordinals, because the set of codes of computable ordinals is badly non-computably-enumerable." I was going to criticize this paper as crankery in the vein of Penrose, but first I thought I'd just compute all possible ordinals in my brain to make sure I'm a general intelligence. brb.
  - xamuel 6y ago
    Hi, thanks for looking at my paper. If you're interested in the relation between Lucas-Penrose stuff and enumeratability of ordinal codes, you might like I.J. Good (1969), "Godel's theorem is a red herring" (2 pages). Can you elaborate on what it is about my paper that strikes you as crankery? I'm a fan of yours so it would be much appreciated.
    - gwern 6y ago
      Sadly, I was only able to count through a vanishingly small subset of the reals in a finite time, and therefore am not a general intelligence, and so it would be foolish of me, a machine made of a handful of atoms, to try to criticize this paper. It sure would be nice if I could appreciate music, but you've proven that's impossible, so it is what it is.
      - xamuel 6y ago
        Whether man is machine can't be trivially answered using a few handwavy applications of Godel's theorem nor of observations about the structure of the reals. My paper does not make any attempt to weigh on that question, neither does it claim nor imply that humans have any supernatural powers of enumerating reals or ordinals, and you grossly misrepresent me by implying it does. Rather, my paper is on the less ambitious question of whether the traditional RL model (with its real-valued rewards) accurately captures the full set of reward-giving environments an AGI should be capable of comprehending.
  - YeGoblynQueenne 6y ago
    None, one, many, all. Hah. Still got it.
- sn41 6y ago
  I think you are right in constructing situations where real numbers are inadequate. It is also right that you do not claim that hyperreals or surreals suffice, you are merely pointing out that they may help you to do better than the reals. But I have often wondered - why are people hung up on linear ordering? Why not non-total partial orders? https://en.wikipedia.org/wiki/Ordinal_optimization https://en.wikipedia.org/wiki/Ordinal_optimization Is this insistence on linear orders because of simplistic modeling on the part of cognitive science and AI people, or is there some problem with general partial orders?
  - xamuel 6y ago
    One could certainly contemplate versions of RL with non-linear orderings. I guess the reason people care about linear ordering is because you want the agent to at least understand "this outcome is better than that outcome". How would we hope for a good nonlinear-RL agent to behave in an environment with 2 buttons, one of which always gives reward X, and the other of which always gives reward Y, where X and Y are incomparable?
    - sn41 6y ago
      That's understandable. But many decisions in life are like that. Do you want to be close to your roots and your parents, or do you want a high-flying career in a remote city? Choices involve sacrifices as well as gains, and many meaningful outcomes are incomparable among themselves.
lostmsu 6y ago
The argument is totally flawed and therefore BS. It rests on a clearly wrong assumption, that you can't represent number sets with higher cardinality (there are more real numbers, than natural numbers) using the sets of lower cardinality. You can use natural numbers to represent real numbers with arbitrary precision. You can also use natural numbers to represent non-Archimedian number systems.
- xamuel 6y ago
  Yes you can, but said representations will necessarily be misleading. It's illuminating to consider Big-O notations: why don't we "simplify", since real numbers are so much easier, why don't we declare, e.g., that O(n) is "1", O(n^2) is "2", etc.? Well, then what should O(2^n) be? A million, perhaps? But then what about O(n^1000000)? To be consistent, you'd have to say O(n^1000000) was something below a million, since that complexity level is below O(2^n). And then whatever you chose for that, that would constrain you to pack O(n^billion), O(n^trillion), etc., into smaller and smaller ranges of possible numbers. You'd inevitably be forced to eventually assign almost identical numbers to, say, O(n^(2^100)) and O(n^(2^999)), which is misleading because those are not at all close to each other in complexity.
  - lostmsu 6y ago
    I don't see how representation of floats by natural numbers that we all use on daily basis is misleading. In fact, your entire comment is just an ordered sequence of natural numbers, and it does not seem to be very misleading (though it tried to trick). Besides, the article is purposed to be a rigorous mathematical proof of current representations in RL being unsuitable for AGI, but I haven't seen a Definition for "misleading" there.
    - sebastos 6y ago
      ?? Current reinforcement learning algorithms don't score an action by producing a line of text that is read by a human. It assigns a value by writing a floating point number into a register on the computer somewhere. If it were otherwise, and there was some kind of interpretation unit that was needed to compare two rewards and decide which one was bigger, then hey, you're not using the reals for your rewards anymore, just like the paper suggested you'd have to.
lasagnaphil 6y ago
Although I have only skimmed the paper, I think it's kinda trying to say (although someone with a better mathematical background than me might poke me for this) that the reward hypothesis (http://www.incompleteideas.net/rlai.cs.ualberta.ca/RLAI/rewardhypothesis.html http://www.incompleteideas.net/rlai.cs.ualberta.ca/RLAI/rewa...) - the notion that every goal or purpose can be framed as the maximization of a real-valued function - isn't really applicable for most of the time. This is quite intuitively agreeable even without the math - do we really think that the many things we do in our lives were perfromed to optimize an "oracle" loss function? Our human mind is comprised of ridiculously complex systems of neurons and cells that generates a variety of emergent behaviors, and saying that those emergent behaviors are actually a solution of a very complex optimization problem is very, very bold. Often the reward functions are just abstractions of what we perceive (although they aren't entirely useless - keep in mind that all models are wrong but some are useful). Although the paper is trying to say that the real number system isn't robust enough to express the goal/purpose of more complicated, "abstract" tasks, it speculates that a higher-order number system (such as the hyperreal or surreal numbers) would be able to achieve this. I currently disagree with this view - I view of "intelligence" as we know of today more as emergent phenomena of complex systems of autonomous agents (in the case of human intelligence, the emergent phenomena of neurons and other cells interacting with the external world), but that's a topic for another day.
- xamuel 6y ago
  I think you understood the basic gist of the paper quite well, that's a good way of describing it, and thanks for the link. >it speculates that a higher-order number system (such as the hyperreal or surreal numbers) would be able to achieve this I didn't mean to give that impression, sorry if it came off that way. Rather, what I say is that those number systems don't suffer the particular flaw that the real numbers suffer. There might still be other flaws. That's why in the beginning of Section 4 I wrote: "There are at least two potential ways to change RL so as to make it applicable to such tasks and, thus, at least potentially capable of leading to AGI. Of course, there is no guarantee that removing the roadblock in this paper will cause RL to lead to AGI. There might be other roadblocks besides the inadequate reward number system"
lacker 6y ago
TLDR: reinforcement learning cannot handle AGI, because reinforcement learning rewards must be finite, but a true artificial intelligence could reason about infinite numbers. I think this is complete nonsense. Humans don't receive infinite rewards, either, but we still think about infinite numbers. We typically think about infinite numbers in terms of finite representations, like finite proofs about their properties. Reinforcement learning could in theory do the same thing. The real reason that traditional reinforcement learning will not yield AGI is that the model is too limited. The current state of the art in reinforcement learning relies on representing the entire world and your entire strategy as a single vector. For most RL tasks we haven't even gotten deep networks working; RL is somewhat behind areas like vision or natural language processing. The way forward isn't to worry about infinite numbers, it's to develop better architectures that let RL solve more problems.
- xamuel 6y ago
  >Humans don't receive infinite rewards Maybe you're right, but I don't think it's as obvious as you imply. Consider this question. Can there exist hypothetical rewards x1,x2,x3,..., each one of which is significantly better than the previous, and at the same time another hypothetical reward y such that y is significantly better than each x_i? If such rewards can exist, and if "significantly better" implies "at least +1 better", then that necessarily means y must be infinite. Now, how would you argue that such rewards can't exist? If you say, "because then y would have to be infinite", then you're arguing in circles, assuming what you want to prove in the first place. It might well be that such rewards indeed can't exist, but if so, the argument would have to be different, and I suspect nontrivial.
  - lostmsu 6y ago
    Another problem with your paper is related to this one: AGI is defined as a program, that can solve any task a human can. You haven't showed, that humans are capable of solving arbitrary tasks with transfinite rewards, therefore you have not demonstrated, that (even potential) inability to solve them within current framework implies inability to create AGI in it.
  - 6y ago
bra-ket 6y ago
I applaud the effort but the problem with RL as a model of learning is in the definition of RL itself. The idea of using "rewards" as a primary learning mechanism and a path to actual cognition is just wrong, full stop. It's a wrong level of abstraction and is too wasteful in energy spent. Looking at it from CogSci perspective it is essentially an offshoot of behaviorism, using a coarse and extremely inefficient model of learning as reward and punishment, iterative trial and error process. This 'Skinnerism' has been discredited in cognitive psychology decades ago and makes absolutely no biological sense whatsoever for the simple reason that any organism trying to adapt in this way will be eaten by predators before minimizing its "error function" sufficiently. Living learning organisms have limited resources (energy and time), and they cut the search space drastically through shortcuts and heuristics and hardcoded biases instead of doing some kind of brute force optimization. This is the case where computational efficiency [1] comes first and sets the constraints by which cognitive apparatus needs to be developed. As for actual cognition models a good place to start is not ML/AI field (which tends to getting stuck in local minima as a whole), but state-of-the-art cognitive psychology, and may be looking at research in "distributional semantics", "concept spaces", "sparse representations", "small-world networks" and "learning and memory" neuroscience. You'd be surprised how much knowledge we gained about the mind since those RL & ANN models developed in the 1940s. [1] https://www.amazon.com/Circuits-Mind-Leslie-G-Valiant/dp/0195126688 https://www.amazon.com/Circuits-Mind-Leslie-G-Valiant/dp/019...
- anotherProd 6y ago
  As someone who's a software engineer (not data scientist) but is interested in consciousness and by extension AGI and dabbled in some ML algorithms, I find it surprising how often I see the sentiments of AGI being possible or impossible using some sort of algorithm. Obviously I could be missing some great breadth and depth of research (there's definitely a lot I don't know) but from what I've read "we have no idea" is a pretty accurate description with how far we've come when it comes to consciousness, and I would imagine even less for the newer field of AI/AGI (consciousness has been around for a while P: and our theories have mostly sidestepped this real world phenomenon). > "The idea of using "rewards" as a learning mechanism and a path to actual cognition is just wrong, full stop." This to me is a huge red flag (mostly of ego/hubris). I think if we rephrased the goal to not talk about "AGI" and maybe around quantitative things like the things you've listed ("computational efficiency", likelihood of being stuck in local minimimas, etc) then I'd happily concede that we should be looking at "X" and not "Y" but unless I've missed something, again likely, when we're talking about AGI, we're talking about consciousness (epiphenomenon that come about through physical/deterministic interactions). A quick way to gut check myself here is twisting what you state is not a good place to start "ML/AI field... gets stuck in local minima" and ask myself is it possible that local minima (which we consider "bad" for current/traditional tasks) could be necessary for consciousness ? I think the widely accepted answer to this is currently "We don't know". If I think that achieving AGI is going to be similar to what the algorithms and architecture we currently use (where the likelihood of being stuck in a local minima is something we can look at) then sure, your opinions stand. But that is just a guess and unless I'm mistaken AGI hasn't been achieved because we don't know how to do it. This isn't to say that we should have 100% of the data before making strong judgements like this about a subject. It's just that the subject of "consciousness" is a big one (I'd say THE big one) so making such strong statements about something we know we don't know much about is interesting. <- this is where I get flashbacks to SE world where a missing piece of data can really throw you off or leads to wrong assumptions and when I think about consciousness we know we don't know a lot.
LatteLazy 6y ago
People love talking about intelligence but I'm yet to see a measurable definition. The best anyone can manage is "you know it when you see it" (like porn). Am I missing something or is this all just a dark road to an unknown destination (possibly just a dead end?)
- PeterisP 6y ago
  One approach is by Legg&Hutter; "Intelligence measures an agent's ability to achieve goals in a wide range of environments", which they also try to formalize in https://zoo.cs.yale.edu/classes/cs671/12f/12f-papers/legg+hutter-universal.pdf https://zoo.cs.yale.edu/classes/cs671/12f/12f-papers/legg+hu... This goes hand in hand with a functional definition of knowledge, where we judge whether a system or an agent "really knows" something through measuring success or failure in a variety of scenarios where knowing that thing (and properly applying that knowledge) is necessary to make an effective decision. Or, quoting Forrest Gump, stupid is as stupid does.
- gnulinux 6y ago
  The value of definability is moot at best imho. "Mathematics" is also not easy to define, there are volumes philosophers of mathematics discussing this, some even say "mathematics is what mathematicians find interesting" (similar to your porn example) but this doesn't stop us from studying mathematics. Same goes for science e.g.. People like Popper or Kuhn spent a lot of mental cycles arguing what is and what is not science, yet people still do science every day without reading them. In some ways this way of thinking is too meta. In order to be a good mathematician, it is not necessary to understand the nature of mathematics from an outside perspective. This view can be very useful e.g. if you're working on foundations, but that doesn't mean before being good at mathematics one must be good at understanding the nature of mathematics. That seems like the job of philosophers, not mathematicians. (Well, sometimes the set has intersections, e.g. Brouwer, Hilbert, Godel and Penrose (theo. physicist) wrote some works on phil. of math). EDIT: To express this slightly more formally: in order to understand a theory of a model, you do not need to understand the model comprehensively. You can be an expert in intelligence science by studying the falsifiable and predictive theories of intelligence science without understanding the nature of intelligence itself.
corey_moncure 6y ago
We have to make machines feel pain
- Smoosh 6y ago
  I've long thought that to reproduce animal-like intelligence artificially, you need to simulate an environment and stimuli, some unequivocally positive and some unequivocally negative. Of course AGI does not need to be "animal-like intelligence".
- heimatau 6y ago
  How do you supposed this is to happen? Some just call it a cost function.
deleted 6y ago
[deleted]
ggm 6y ago
I like reductionist maths counter-proofs based on "x cannot contain y", but I try to be skeptical they apply, because it is plain real mathematicians can reason about infinite things, from finite symbols, as chains of symbols. the "cannot contain" set includes things I can state, but not enumerate. I'm skeptical about AGI anyway. The proof is unnecessary, I tend to an even more reductionist model: Our lack of understanding where intelligence is, in the brain, goes to our failure to model it. "morally" making claims "its alive" based on a lack of understanding, is a bit like chemistry by alchemy. If you don't know why, you didn't make it any more than making a baby would have. Feynman said it better: lots of physics proofs are built on partial models of sub-atomics, which if you ask questions about become unknowables too. Its turtles-all-the-way-down stuff. traditional reinforcement learning will probably not yield AGI, any more than any current method, based on not understanding GI, and until we understand GI, I do not believe any connectionist, or learning model will derive it.
musicale 6y ago
Pretty sure the IRS would figure out a way to defeat any sufficiently advanced AI system.
stosto88 6y ago
Yeah of course not, computers today as we know them will never lead to agi.
jula432vdf 6y ago
So cockroaches have most probably not a big neural network, but they are able to do lots of stuff, including reproduction. I think this kind of machine intelligence is already at reach of currents models, or we are close to be able to make an "e-cockroach". I think we'll learn lots of stuff just seconds after having put in the world that kind of limited artificial intelligence. And then the models, through sensors, will have the entire world at reach to begin auto-improvement tasks. And this would solve the issue "we're just training the models with very limited data, how could they evolve faster than millions of years?" I think the FAANG, the big players, realized this limitation a LOT time ago (10 years maybe). And they are already trying some things to solve the limited data issue, giving their models all the information they can extract from cellphones, the current iteration of "massive network of sensors to train BIG - multiple, almost hidden from the public - models. If I have to bet, I'd bet the whole information is being stored to be "replayed" when whole new more advanced models emerge eventually.
- deleted 6y ago
  [deleted]

82 points by xamuel 6y ago

Author here. One particularly topical observation (topical because HN has recently featured discussions about ways to merge statistical and symbolic approaches to AI), from Section 4.2.3: certain cutting-edge number systems such as Conway's "surreal numbers" are so sophisticated that they require lots of symbolic logic just to do basic operations with. Of course this makes these number systems hard to work with, but they do not suffer the flaws pointed out in the paper, which the easier-to-work-with real numbers suffer. Since surreal numbers inherently require symbolic logical methods to work with, it follows that any sort of statistics-based RL agent for environments with surreal number rewards, would automatically combine symbolic logic and statistics.

nx2059 6y ago
The article was pretty over my head, but does your argument hold to the various augmented neural net systems such as Neural Turing Machines, or Differentiable Neural Computers?
- xamuel 6y ago
  The argument isn't so much about the type of agent (which I think is what Neural Turing Machines etc. are about), it's about the type of environment. In traditional Reinforcement Learning, environments give real-number-valued rewards (or even rational-number-valued rewards which is even more constrained). Presumably this was a decision that was made with hardly a second thought because real numbers are most familiar to people... but such "appeals to familiarity" are totally irrelevant for such an alien field as AGI :) The point of the paper is that a genuine AGI should be able to comprehend environments that involve rewards with a more sophisticated structure than can be accurately represented using real numbers.
  - ssivark 6y ago
    IIUC, the claim is that the very idea of a (real valued) “objective function” to be “optimized” is broken?
    - xamuel 6y ago
      Broken in the sense that it's not flexible enough to apply to all conceivable environments a genuine AGI could navigate, without misleading that AGI. But I should stress that real number objective functions are probably fine for many specific interesting environments, I'm not trying to say that real number objective functions are useless. Just that they aren't flexible enough to cover all environments :)
      - ssivark 6y ago
        Fair enough. It would be interesting/instructive to construct (relatively simple) examples where we can see that they’re broken :-)
        xamuel 6y ago
        Easy to come up with examples using exotic money-related constructions. Suppose there's something called a "superdollar". If you have a superdollar, you can use it to create an arbitrary number of dollars for yourself, any time you want, which you can trade for goods and services. If you want, you can also trade the superdollar itself. Now picture an environment with two buttons, one of which always rewards you one dollar, and the other of which always rewards you one superdollar. Shoe-horning this environment into traditional RL, you'd have to assign the superdollar button some finite reward, say a million. But then you would mislead the traditional-RL-agent into thinking a million dollars was as good as one superdollar, which clearly is not true.
  - IshKebab 6y ago
    But isn't the sophisticated structure what leads to the real number. If you change the rewards to something more complex surely you still have to pick between actions and at some point you'll have to evaluate which one is "better" and I can't see why you couldn't use real numbers to represent utility. I mean, humans are general intelligences, and you can translate pretty much any human reward into money, which is a real number. The paper is super long though. Maybe someone can give a TL;DR that makes sense.
TTPrograms 6y ago
I think you are hiding a key assumption - that an AGI must be capable of performing optimally on a task with Archimedean measure. This is a very strong assumption - performing eps-close to optimal may certainly fall in the definition of AGI, and this might be achieved by a non-Archimedean approximation. Defining AGI as performing optimally on any set of tasks is problematic from a computational theory perspective in general - even with real reward signals. Furthermore, infinite rewards are not compatible with human behavior. Humans never optimize for a single event at infinite expense w.r.t. other goals.
- Animats 6y ago
  That an AGI must be capable of performing optimally on a task with Archimedean measure. If that's correct, this may be just one of those many problems where the optimal solution is far harder than a near-optimal solution. Examples include linear programming and the traveling salesman problem, where a true optimum is NP-hard to find, but you can get very close with far less work.
- xamuel 6y ago
  Hi, thanks for looking at my paper. I do not assume that an AGI must be capable of performing optimally on all tasks in general--indeed, that's quite impossible. When measuring the performance of RL agents, one must come up with some way of aggregating performance across many environments, but that's beside the point of this paper. The point of this paper is that if you're forced to use real numbers as rewards, you can't even communicate all environments to the agent without misleading the agent. Whether the agent could perform well or poorly in the environments once communicated, is beside the point.
bra-ket 6y ago
do your claims apply in complex-valued RL context?
- xamuel 6y ago
  I'm surprised to see that that's actually a thing people write about. I'm afraid I can't answer your question right now, as I have no idea why anyone would do that (as opposed to, say, vector-valued rewards---i.e., why is the ability to complex-multiply two complex rewards relevant, as opposed to merely comparing separate components of rewards from R^2). I'll have to read up on the subject :)
gwern 6y ago
"Assuming AGI agents are Turing computable, no individual AGI can possibly comprehend codes for all computable ordinals, because the set of codes of computable ordinals is badly non-computably-enumerable." I was going to criticize this paper as crankery in the vein of Penrose, but first I thought I'd just compute all possible ordinals in my brain to make sure I'm a general intelligence. brb.
- xamuel 6y ago
  Hi, thanks for looking at my paper. If you're interested in the relation between Lucas-Penrose stuff and enumeratability of ordinal codes, you might like I.J. Good (1969), "Godel's theorem is a red herring" (2 pages). Can you elaborate on what it is about my paper that strikes you as crankery? I'm a fan of yours so it would be much appreciated.
  - gwern 6y ago
    Sadly, I was only able to count through a vanishingly small subset of the reals in a finite time, and therefore am not a general intelligence, and so it would be foolish of me, a machine made of a handful of atoms, to try to criticize this paper. It sure would be nice if I could appreciate music, but you've proven that's impossible, so it is what it is.
    - xamuel 6y ago
      Whether man is machine can't be trivially answered using a few handwavy applications of Godel's theorem nor of observations about the structure of the reals. My paper does not make any attempt to weigh on that question, neither does it claim nor imply that humans have any supernatural powers of enumerating reals or ordinals, and you grossly misrepresent me by implying it does. Rather, my paper is on the less ambitious question of whether the traditional RL model (with its real-valued rewards) accurately captures the full set of reward-giving environments an AGI should be capable of comprehending.
- YeGoblynQueenne 6y ago
  None, one, many, all. Hah. Still got it.
sn41 6y ago
I think you are right in constructing situations where real numbers are inadequate. It is also right that you do not claim that hyperreals or surreals suffice, you are merely pointing out that they may help you to do better than the reals. But I have often wondered - why are people hung up on linear ordering? Why not non-total partial orders? https://en.wikipedia.org/wiki/Ordinal_optimization https://en.wikipedia.org/wiki/Ordinal_optimization Is this insistence on linear orders because of simplistic modeling on the part of cognitive science and AI people, or is there some problem with general partial orders?
- xamuel 6y ago
  One could certainly contemplate versions of RL with non-linear orderings. I guess the reason people care about linear ordering is because you want the agent to at least understand "this outcome is better than that outcome". How would we hope for a good nonlinear-RL agent to behave in an environment with 2 buttons, one of which always gives reward X, and the other of which always gives reward Y, where X and Y are incomparable?
  - sn41 6y ago
    That's understandable. But many decisions in life are like that. Do you want to be close to your roots and your parents, or do you want a high-flying career in a remote city? Choices involve sacrifices as well as gains, and many meaningful outcomes are incomparable among themselves.

lostmsu 6y ago

The argument is totally flawed and therefore BS. It rests on a clearly wrong assumption, that you can't represent number sets with higher cardinality (there are more real numbers, than natural numbers) using the sets of lower cardinality. You can use natural numbers to represent real numbers with arbitrary precision. You can also use natural numbers to represent non-Archimedian number systems.

xamuel 6y ago
Yes you can, but said representations will necessarily be misleading. It's illuminating to consider Big-O notations: why don't we "simplify", since real numbers are so much easier, why don't we declare, e.g., that O(n) is "1", O(n^2) is "2", etc.? Well, then what should O(2^n) be? A million, perhaps? But then what about O(n^1000000)? To be consistent, you'd have to say O(n^1000000) was something below a million, since that complexity level is below O(2^n). And then whatever you chose for that, that would constrain you to pack O(n^billion), O(n^trillion), etc., into smaller and smaller ranges of possible numbers. You'd inevitably be forced to eventually assign almost identical numbers to, say, O(n^(2^100)) and O(n^(2^999)), which is misleading because those are not at all close to each other in complexity.
- lostmsu 6y ago
  I don't see how representation of floats by natural numbers that we all use on daily basis is misleading. In fact, your entire comment is just an ordered sequence of natural numbers, and it does not seem to be very misleading (though it tried to trick). Besides, the article is purposed to be a rigorous mathematical proof of current representations in RL being unsuitable for AGI, but I haven't seen a Definition for "misleading" there.
  - sebastos 6y ago
    ?? Current reinforcement learning algorithms don't score an action by producing a line of text that is read by a human. It assigns a value by writing a floating point number into a register on the computer somewhere. If it were otherwise, and there was some kind of interpretation unit that was needed to compare two rewards and decide which one was bigger, then hey, you're not using the reals for your rewards anymore, just like the paper suggested you'd have to.

lasagnaphil 6y ago

Although I have only skimmed the paper, I think it's kinda trying to say (although someone with a better mathematical background than me might poke me for this) that the reward hypothesis (http://www.incompleteideas.net/rlai.cs.ualberta.ca/RLAI/rewardhypothesis.html http://www.incompleteideas.net/rlai.cs.ualberta.ca/RLAI/rewa...) - the notion that every goal or purpose can be framed as the maximization of a real-valued function - isn't really applicable for most of the time. This is quite intuitively agreeable even without the math - do we really think that the many things we do in our lives were perfromed to optimize an "oracle" loss function? Our human mind is comprised of ridiculously complex systems of neurons and cells that generates a variety of emergent behaviors, and saying that those emergent behaviors are actually a solution of a very complex optimization problem is very, very bold. Often the reward functions are just abstractions of what we perceive (although they aren't entirely useless - keep in mind that all models are wrong but some are useful). Although the paper is trying to say that the real number system isn't robust enough to express the goal/purpose of more complicated, "abstract" tasks, it speculates that a higher-order number system (such as the hyperreal or surreal numbers) would be able to achieve this. I currently disagree with this view - I view of "intelligence" as we know of today more as emergent phenomena of complex systems of autonomous agents (in the case of human intelligence, the emergent phenomena of neurons and other cells interacting with the external world), but that's a topic for another day.

xamuel 6y ago
I think you understood the basic gist of the paper quite well, that's a good way of describing it, and thanks for the link. >it speculates that a higher-order number system (such as the hyperreal or surreal numbers) would be able to achieve this I didn't mean to give that impression, sorry if it came off that way. Rather, what I say is that those number systems don't suffer the particular flaw that the real numbers suffer. There might still be other flaws. That's why in the beginning of Section 4 I wrote: "There are at least two potential ways to change RL so as to make it applicable to such tasks and, thus, at least potentially capable of leading to AGI. Of course, there is no guarantee that removing the roadblock in this paper will cause RL to lead to AGI. There might be other roadblocks besides the inadequate reward number system"

lacker 6y ago

TLDR: reinforcement learning cannot handle AGI, because reinforcement learning rewards must be finite, but a true artificial intelligence could reason about infinite numbers. I think this is complete nonsense. Humans don't receive infinite rewards, either, but we still think about infinite numbers. We typically think about infinite numbers in terms of finite representations, like finite proofs about their properties. Reinforcement learning could in theory do the same thing. The real reason that traditional reinforcement learning will not yield AGI is that the model is too limited. The current state of the art in reinforcement learning relies on representing the entire world and your entire strategy as a single vector. For most RL tasks we haven't even gotten deep networks working; RL is somewhat behind areas like vision or natural language processing. The way forward isn't to worry about infinite numbers, it's to develop better architectures that let RL solve more problems.

xamuel 6y ago
>Humans don't receive infinite rewards Maybe you're right, but I don't think it's as obvious as you imply. Consider this question. Can there exist hypothetical rewards x1,x2,x3,..., each one of which is significantly better than the previous, and at the same time another hypothetical reward y such that y is significantly better than each x_i? If such rewards can exist, and if "significantly better" implies "at least +1 better", then that necessarily means y must be infinite. Now, how would you argue that such rewards can't exist? If you say, "because then y would have to be infinite", then you're arguing in circles, assuming what you want to prove in the first place. It might well be that such rewards indeed can't exist, but if so, the argument would have to be different, and I suspect nontrivial.
- lostmsu 6y ago
  Another problem with your paper is related to this one: AGI is defined as a program, that can solve any task a human can. You haven't showed, that humans are capable of solving arbitrary tasks with transfinite rewards, therefore you have not demonstrated, that (even potential) inability to solve them within current framework implies inability to create AGI in it.
- 6y ago

bra-ket 6y ago

I applaud the effort but the problem with RL as a model of learning is in the definition of RL itself. The idea of using "rewards" as a primary learning mechanism and a path to actual cognition is just wrong, full stop. It's a wrong level of abstraction and is too wasteful in energy spent. Looking at it from CogSci perspective it is essentially an offshoot of behaviorism, using a coarse and extremely inefficient model of learning as reward and punishment, iterative trial and error process. This 'Skinnerism' has been discredited in cognitive psychology decades ago and makes absolutely no biological sense whatsoever for the simple reason that any organism trying to adapt in this way will be eaten by predators before minimizing its "error function" sufficiently. Living learning organisms have limited resources (energy and time), and they cut the search space drastically through shortcuts and heuristics and hardcoded biases instead of doing some kind of brute force optimization. This is the case where computational efficiency [1] comes first and sets the constraints by which cognitive apparatus needs to be developed. As for actual cognition models a good place to start is not ML/AI field (which tends to getting stuck in local minima as a whole), but state-of-the-art cognitive psychology, and may be looking at research in "distributional semantics", "concept spaces", "sparse representations", "small-world networks" and "learning and memory" neuroscience. You'd be surprised how much knowledge we gained about the mind since those RL & ANN models developed in the 1940s. [1] https://www.amazon.com/Circuits-Mind-Leslie-G-Valiant/dp/0195126688 https://www.amazon.com/Circuits-Mind-Leslie-G-Valiant/dp/019...

anotherProd 6y ago
As someone who's a software engineer (not data scientist) but is interested in consciousness and by extension AGI and dabbled in some ML algorithms, I find it surprising how often I see the sentiments of AGI being possible or impossible using some sort of algorithm. Obviously I could be missing some great breadth and depth of research (there's definitely a lot I don't know) but from what I've read "we have no idea" is a pretty accurate description with how far we've come when it comes to consciousness, and I would imagine even less for the newer field of AI/AGI (consciousness has been around for a while P: and our theories have mostly sidestepped this real world phenomenon). > "The idea of using "rewards" as a learning mechanism and a path to actual cognition is just wrong, full stop." This to me is a huge red flag (mostly of ego/hubris). I think if we rephrased the goal to not talk about "AGI" and maybe around quantitative things like the things you've listed ("computational efficiency", likelihood of being stuck in local minimimas, etc) then I'd happily concede that we should be looking at "X" and not "Y" but unless I've missed something, again likely, when we're talking about AGI, we're talking about consciousness (epiphenomenon that come about through physical/deterministic interactions). A quick way to gut check myself here is twisting what you state is not a good place to start "ML/AI field... gets stuck in local minima" and ask myself is it possible that local minima (which we consider "bad" for current/traditional tasks) could be necessary for consciousness ? I think the widely accepted answer to this is currently "We don't know". If I think that achieving AGI is going to be similar to what the algorithms and architecture we currently use (where the likelihood of being stuck in a local minima is something we can look at) then sure, your opinions stand. But that is just a guess and unless I'm mistaken AGI hasn't been achieved because we don't know how to do it. This isn't to say that we should have 100% of the data before making strong judgements like this about a subject. It's just that the subject of "consciousness" is a big one (I'd say THE big one) so making such strong statements about something we know we don't know much about is interesting. <- this is where I get flashbacks to SE world where a missing piece of data can really throw you off or leads to wrong assumptions and when I think about consciousness we know we don't know a lot.

LatteLazy 6y ago

People love talking about intelligence but I'm yet to see a measurable definition. The best anyone can manage is "you know it when you see it" (like porn). Am I missing something or is this all just a dark road to an unknown destination (possibly just a dead end?)

PeterisP 6y ago
One approach is by Legg&Hutter; "Intelligence measures an agent's ability to achieve goals in a wide range of environments", which they also try to formalize in https://zoo.cs.yale.edu/classes/cs671/12f/12f-papers/legg+hutter-universal.pdf https://zoo.cs.yale.edu/classes/cs671/12f/12f-papers/legg+hu... This goes hand in hand with a functional definition of knowledge, where we judge whether a system or an agent "really knows" something through measuring success or failure in a variety of scenarios where knowing that thing (and properly applying that knowledge) is necessary to make an effective decision. Or, quoting Forrest Gump, stupid is as stupid does.
gnulinux 6y ago
The value of definability is moot at best imho. "Mathematics" is also not easy to define, there are volumes philosophers of mathematics discussing this, some even say "mathematics is what mathematicians find interesting" (similar to your porn example) but this doesn't stop us from studying mathematics. Same goes for science e.g.. People like Popper or Kuhn spent a lot of mental cycles arguing what is and what is not science, yet people still do science every day without reading them. In some ways this way of thinking is too meta. In order to be a good mathematician, it is not necessary to understand the nature of mathematics from an outside perspective. This view can be very useful e.g. if you're working on foundations, but that doesn't mean before being good at mathematics one must be good at understanding the nature of mathematics. That seems like the job of philosophers, not mathematicians. (Well, sometimes the set has intersections, e.g. Brouwer, Hilbert, Godel and Penrose (theo. physicist) wrote some works on phil. of math). EDIT: To express this slightly more formally: in order to understand a theory of a model, you do not need to understand the model comprehensively. You can be an expert in intelligence science by studying the falsifiable and predictive theories of intelligence science without understanding the nature of intelligence itself.

corey_moncure 6y ago

We have to make machines feel pain

Smoosh 6y ago
I've long thought that to reproduce animal-like intelligence artificially, you need to simulate an environment and stimuli, some unequivocally positive and some unequivocally negative. Of course AGI does not need to be "animal-like intelligence".
heimatau 6y ago
How do you supposed this is to happen? Some just call it a cost function.

deleted 6y ago

[deleted]

ggm 6y ago

I like reductionist maths counter-proofs based on "x cannot contain y", but I try to be skeptical they apply, because it is plain real mathematicians can reason about infinite things, from finite symbols, as chains of symbols. the "cannot contain" set includes things I can state, but not enumerate. I'm skeptical about AGI anyway. The proof is unnecessary, I tend to an even more reductionist model: Our lack of understanding where intelligence is, in the brain, goes to our failure to model it. "morally" making claims "its alive" based on a lack of understanding, is a bit like chemistry by alchemy. If you don't know why, you didn't make it any more than making a baby would have. Feynman said it better: lots of physics proofs are built on partial models of sub-atomics, which if you ask questions about become unknowables too. Its turtles-all-the-way-down stuff. traditional reinforcement learning will probably not yield AGI, any more than any current method, based on not understanding GI, and until we understand GI, I do not believe any connectionist, or learning model will derive it.

musicale 6y ago

Pretty sure the IRS would figure out a way to defeat any sufficiently advanced AI system.

stosto88 6y ago

Yeah of course not, computers today as we know them will never lead to agi.

jula432vdf 6y ago

So cockroaches have most probably not a big neural network, but they are able to do lots of stuff, including reproduction. I think this kind of machine intelligence is already at reach of currents models, or we are close to be able to make an "e-cockroach". I think we'll learn lots of stuff just seconds after having put in the world that kind of limited artificial intelligence. And then the models, through sensors, will have the entire world at reach to begin auto-improvement tasks. And this would solve the issue "we're just training the models with very limited data, how could they evolve faster than millions of years?" I think the FAANG, the big players, realized this limitation a LOT time ago (10 years maybe). And they are already trying some things to solve the limited data issue, giving their models all the information they can extract from cellphones, the current iteration of "massive network of sensors to train BIG - multiple, almost hidden from the public - models. If I have to bet, I'd bet the whole information is being stored to be "replayed" when whole new more advanced models emerge eventually.

deleted 6y ago
[deleted]