5 ms·

TLDR: reinforcement learning cannot handle AGI, because reinforcement learning rewards must be finite, but a true artificial intelligence could reason about inf

by lacker 6y ago

TLDR: reinforcement learning cannot handle AGI, because reinforcement learning rewards must be finite, but a true artificial intelligence could reason about infinite numbers. I think this is complete nonsense. Humans don't receive infinite rewards, either, but we still think about infinite numbers. We typically think about infinite numbers in terms of finite representations, like finite proofs about their properties. Reinforcement learning could in theory do the same thing. The real reason that traditional reinforcement learning will not yield AGI is that the model is too limited. The current state of the art in reinforcement learning relies on representing the entire world and your entire strategy as a single vector. For most RL tasks we haven't even gotten deep networks working; RL is somewhat behind areas like vision or natural language processing. The way forward isn't to worry about infinite numbers, it's to develop better architectures that let RL solve more problems.

xamuel 6y ago
>Humans don't receive infinite rewards Maybe you're right, but I don't think it's as obvious as you imply. Consider this question. Can there exist hypothetical rewards x1,x2,x3,..., each one of which is significantly better than the previous, and at the same time another hypothetical reward y such that y is significantly better than each x_i? If such rewards can exist, and if "significantly better" implies "at least +1 better", then that necessarily means y must be infinite. Now, how would you argue that such rewards can't exist? If you say, "because then y would have to be infinite", then you're arguing in circles, assuming what you want to prove in the first place. It might well be that such rewards indeed can't exist, but if so, the argument would have to be different, and I suspect nontrivial.
- lostmsu 6y ago
  Another problem with your paper is related to this one: AGI is defined as a program, that can solve any task a human can. You haven't showed, that humans are capable of solving arbitrary tasks with transfinite rewards, therefore you have not demonstrated, that (even potential) inability to solve them within current framework implies inability to create AGI in it.
- PeterisP 6y ago
  Humans don't receive infinite rewards because any "reward system" in our brain is implemented by receiving a finite amount of electrochemical pleasure signals over a finite amount of time. Human-equivalent minds can obviously be implemented on top a framework that does not have inherently infinite or infinitely divisible values, because human minds are implemented on a top of a substrate that uses a finite amount of discrete neurotransmitters to do everything, and manages to work just fine. If human minds can reason about infinite concepts using "hardware" where all the neural signals used to represent that concept are finite, then why should it be impossible a reinforcement learning system learn to reason about infinite concepts while using only finite numbers to form that representation?
  - xamuel 6y ago
    >Humans don't receive infinite rewards because any "reward system" in our brain is implemented by receiving a finite amount of electrochemical pleasure signals over a finite amount of time. This is like saying computers can't represent infinity because they have only finitely many bytes. Suppose the treasury rewarded you a "superdollar", which is a special object that allows you to create any number of dollars that you want, on demand, as many times as you want. How many dollars would you say this superdollar is worth? Obviously, no finite number of dollars would be worth that one superdollar. The human mind can certainly understand the relative value of a superdollar vs. any number of dollars. That the human mind is implemented through finite electrochemical processes is irrelevant.
    - PeterisP 6y ago
      The worth of a superdollar is equal to as much dollars as you can spend per second times the seconds you can expect to live. Both numbers are large but finite, the first is bounded by the value of global economy (no matter how many dollars you create, the total purchasing power of these dollars can't grow beyond that), the second is bounded if not to a few hundred years then by the time until the heat death of the universe. In a similar manner, the total amount of reward that you might ever get is capped by the amount of pleasure your brain can perceive at any given moment (which is finite) times your lifespan (which is finite). You might reason and hypothesise about agents perceiving infinite rewards (as we are doing now), but this has nothing to do with the reality of homo sapiens rewards system(s), or, in fact, the rewards system of any agent existing in our physical reality, which is effectively bounded both in time and space.
nx2059 6y ago
So in this paper (https://arxiv.org/pdf/1707.02389.pdf https://arxiv.org/pdf/1707.02389.pdf), (If I'm groking it correctly) Terrance Tao seems to embed a Turing Machine in a vector space. So is using a single vector not a real problem, or am I just not getting it?
sebastos 6y ago
Is that an accurate TL;DR? I'm pretty sure no... It's not saying you need rewards that are infinite in value per se. It's saying that you might have more situations that need to be differentiated from each other than there are real numbers. For instance, you might need to have one set of rewards that map to the real numbers, and then another set of rewards that also map to the real numbers when compared to themselves, but are all considered strictly greater than the first set.
- PeterisP 6y ago
  It's assserting that there's a need to differentiate more situations from each other than there are real numbers, and basing its conclusions on the assertion, but it is not providing a convincing basis why that assertion/assumption/hypothesis is true. The author demonstrates that a system without such a capability would not be able to solve a certain set of problems. However, going from that claim to a claim that this capability is needed for human level AGI is a non-sequitur - there is no evidence that such a capability is needed for human-level intelligence, and there's no evidence (at least not mentioned in the paper) that humans have the exact capability described.
causality0 6y ago
Agreed. In terms of base assumptions, running AGI on computers also puts us at a disadvantage due to multiple layers of abstraction. That is to say, AGI is what organic brains are as opposed to what we're asking computers to do. It's like a complexity difference between building a machine to throw a ball and building one to accurately simulate the physics of throwing a soft rubber ball through turbulent air onto grass. A dragonfly uses just sixteen neurons to take visual input from thousands of ommatidia and use it to track prey in 3D space. How many transistors would we have to use to accomplish the same task? Now scale that up to 86 billion neurons in a human brain. That's the scale of the problem we're looking at after we figure out how to program it.
dumbfoundded 6y ago
Humans do have infinite rewards (or at least penalties). Death is a pretty common one for example.
- PeterisP 6y ago
  We do not treat death as an infinite penalty in our decisionmaking, as evidenced by various cases of people choosing to die for one reason or another, prioritizing the achievement of some other goal (with a finite reward) over survival. People can and do use statements like "death must be avoided at all costs" (it's also mentioned in the paper), but that statement simply is hyperbole which is not exactly literally true description of reality.
drdeca 6y ago
I don't think the numbers really have to be "infinite" exactly (well, I suppose it depends what you mean). Suppose you introduce a data type which consists of an ordered pair of floating point numbers, along with an ordering relation, such that the elements of the data type are ordered lexicographically (i.e. if the first entries differ, then the one with a larger first entry is larger. Otherwise, the one with the larger second entry. If they are equal then they are equal.) If you have an environment which gives rewards of this type, and you want the model which gets the highest reward, you are likely to have an issue if you try to represent the rewards using a single floating point number. (well, you could just use the first entry, and do decently well, but you would lose out a bit on what could be accomplished on the second number.) Of course, because there are, in actuality, only finitely many floating point values of a given precision, you can actually give an enumeration of the values of this type in order, and if you use that enumeration for the rewards, then that could work. However, when we use floating point numbers, we sometimes sorta-pretend that they don't have a finite range of actual-number-values . We sometimes sorta-pretend that they are the actual real numbers (with a little fuzziness and errors tacked on, when we are being careful). We use them in computing "derivatives" and such. And this works pretty well! But if we wanted to use the enumeration of the pairs of floats, if we just treated them as the (bigint) integer index of the pair, we would lose all the nice interpretation that goes with floating point numbers, and be left with only the ordering. The notion of the relative distances between the different values would be lost. The whole "use the derivative of this function (except we are using floating point numbers)" trick stops being applicable. While adding together any number of copies of (0.0 , 1.0) would never surpass (1.0 , -5.0) , that's not really a reason to make it so we can't define the preferences we want to represent, uh, unable to handle that situation.

by lacker 6y ago

xamuel 6y ago

>Humans don't receive infinite rewards Maybe you're right, but I don't think it's as obvious as you imply. Consider this question. Can there exist hypothetical rewards x1,x2,x3,..., each one of which is significantly better than the previous, and at the same time another hypothetical reward y such that y is significantly better than each x_i? If such rewards can exist, and if "significantly better" implies "at least +1 better", then that necessarily means y must be infinite. Now, how would you argue that such rewards can't exist? If you say, "because then y would have to be infinite", then you're arguing in circles, assuming what you want to prove in the first place. It might well be that such rewards indeed can't exist, but if so, the argument would have to be different, and I suspect nontrivial.

lostmsu 6y ago
Another problem with your paper is related to this one: AGI is defined as a program, that can solve any task a human can. You haven't showed, that humans are capable of solving arbitrary tasks with transfinite rewards, therefore you have not demonstrated, that (even potential) inability to solve them within current framework implies inability to create AGI in it.
PeterisP 6y ago
Humans don't receive infinite rewards because any "reward system" in our brain is implemented by receiving a finite amount of electrochemical pleasure signals over a finite amount of time. Human-equivalent minds can obviously be implemented on top a framework that does not have inherently infinite or infinitely divisible values, because human minds are implemented on a top of a substrate that uses a finite amount of discrete neurotransmitters to do everything, and manages to work just fine. If human minds can reason about infinite concepts using "hardware" where all the neural signals used to represent that concept are finite, then why should it be impossible a reinforcement learning system learn to reason about infinite concepts while using only finite numbers to form that representation?
- xamuel 6y ago
  >Humans don't receive infinite rewards because any "reward system" in our brain is implemented by receiving a finite amount of electrochemical pleasure signals over a finite amount of time. This is like saying computers can't represent infinity because they have only finitely many bytes. Suppose the treasury rewarded you a "superdollar", which is a special object that allows you to create any number of dollars that you want, on demand, as many times as you want. How many dollars would you say this superdollar is worth? Obviously, no finite number of dollars would be worth that one superdollar. The human mind can certainly understand the relative value of a superdollar vs. any number of dollars. That the human mind is implemented through finite electrochemical processes is irrelevant.
  - PeterisP 6y ago
    The worth of a superdollar is equal to as much dollars as you can spend per second times the seconds you can expect to live. Both numbers are large but finite, the first is bounded by the value of global economy (no matter how many dollars you create, the total purchasing power of these dollars can't grow beyond that), the second is bounded if not to a few hundred years then by the time until the heat death of the universe. In a similar manner, the total amount of reward that you might ever get is capped by the amount of pleasure your brain can perceive at any given moment (which is finite) times your lifespan (which is finite). You might reason and hypothesise about agents perceiving infinite rewards (as we are doing now), but this has nothing to do with the reality of homo sapiens rewards system(s), or, in fact, the rewards system of any agent existing in our physical reality, which is effectively bounded both in time and space.

nx2059 6y ago

So in this paper (https://arxiv.org/pdf/1707.02389.pdf https://arxiv.org/pdf/1707.02389.pdf), (If I'm groking it correctly) Terrance Tao seems to embed a Turing Machine in a vector space. So is using a single vector not a real problem, or am I just not getting it?

sebastos 6y ago

Is that an accurate TL;DR? I'm pretty sure no... It's not saying you need rewards that are infinite in value per se. It's saying that you might have more situations that need to be differentiated from each other than there are real numbers. For instance, you might need to have one set of rewards that map to the real numbers, and then another set of rewards that also map to the real numbers when compared to themselves, but are all considered strictly greater than the first set.

PeterisP 6y ago
It's assserting that there's a need to differentiate more situations from each other than there are real numbers, and basing its conclusions on the assertion, but it is not providing a convincing basis why that assertion/assumption/hypothesis is true. The author demonstrates that a system without such a capability would not be able to solve a certain set of problems. However, going from that claim to a claim that this capability is needed for human level AGI is a non-sequitur - there is no evidence that such a capability is needed for human-level intelligence, and there's no evidence (at least not mentioned in the paper) that humans have the exact capability described.

causality0 6y ago

Agreed. In terms of base assumptions, running AGI on computers also puts us at a disadvantage due to multiple layers of abstraction. That is to say, AGI is what organic brains are as opposed to what we're asking computers to do. It's like a complexity difference between building a machine to throw a ball and building one to accurately simulate the physics of throwing a soft rubber ball through turbulent air onto grass. A dragonfly uses just sixteen neurons to take visual input from thousands of ommatidia and use it to track prey in 3D space. How many transistors would we have to use to accomplish the same task? Now scale that up to 86 billion neurons in a human brain. That's the scale of the problem we're looking at after we figure out how to program it.

dumbfoundded 6y ago

Humans do have infinite rewards (or at least penalties). Death is a pretty common one for example.

PeterisP 6y ago
We do not treat death as an infinite penalty in our decisionmaking, as evidenced by various cases of people choosing to die for one reason or another, prioritizing the achievement of some other goal (with a finite reward) over survival. People can and do use statements like "death must be avoided at all costs" (it's also mentioned in the paper), but that statement simply is hyperbole which is not exactly literally true description of reality.

drdeca 6y ago

I don't think the numbers really have to be "infinite" exactly (well, I suppose it depends what you mean). Suppose you introduce a data type which consists of an ordered pair of floating point numbers, along with an ordering relation, such that the elements of the data type are ordered lexicographically (i.e. if the first entries differ, then the one with a larger first entry is larger. Otherwise, the one with the larger second entry. If they are equal then they are equal.) If you have an environment which gives rewards of this type, and you want the model which gets the highest reward, you are likely to have an issue if you try to represent the rewards using a single floating point number. (well, you could just use the first entry, and do decently well, but you would lose out a bit on what could be accomplished on the second number.) Of course, because there are, in actuality, only finitely many floating point values of a given precision, you can actually give an enumeration of the values of this type in order, and if you use that enumeration for the rewards, then that could work. However, when we use floating point numbers, we sometimes sorta-pretend that they don't have a finite range of actual-number-values . We sometimes sorta-pretend that they are the actual real numbers (with a little fuzziness and errors tacked on, when we are being careful). We use them in computing "derivatives" and such. And this works pretty well! But if we wanted to use the enumeration of the pairs of floats, if we just treated them as the (bigint) integer index of the pair, we would lose all the nice interpretation that goes with floating point numbers, and be left with only the ordering. The notion of the relative distances between the different values would be lost. The whole "use the derivative of this function (except we are using floating point numbers)" trick stops being applicable. While adding together any number of copies of (0.0 , 1.0) would never surpass (1.0 , -5.0) , that's not really a reason to make it so we can't define the preferences we want to represent, uh, unable to handle that situation.