6 ms·

Reward Is Enough (2021)

21 points by optimalsolver 4y ago

strangattractor 4y ago
Science Direct would think that given that all their reviewers and authors are paid zero dollars but Science Direct turns a profit.
- Thorrez 4y ago
  If you define reward as dollars, then the reviewers and authors receive no reward.
goldenkey 4y ago
I for one, would greatly forgo reward if it were to avoid suffering. I don't think intelligence is simplistic enough to be driven by just a scalar field. I think like physics, we'll likely need two scalars, or a vector, or a spinor, etc. If "-1" is suffering, and "+1" is reward, then they cancel out. In my own experience, suffering and reward do not cancel out. They have to be on different dimensions. What are your thoughts?
- XorNot 4y ago
  In studies it seems to take about 5 positive experiences to undo the mental impact of 1 negative experience. [1] So yes - generally these are not considered equal. From almost every perspective, this also makes sense - trading 1 for 1 doesn't lead to you improving your situation. [1] https://www.seekingbalance.com.au/wp-content/uploads/2016/06/BadStrongerThanGood.pdf https://www.seekingbalance.com.au/wp-content/uploads/2016/06...
  - anon_123g987 4y ago
    The claim here is even stronger, and somewhat different: that suffering and reward are fundamentally different kind of things, and cannot simply undo each other, not at any exchange rate. The difference is qualitative, not quantitative. For example, if you lose your eyesight, it's such a big loss, that most likely nothing can balance it out ever. It has no "five times as good" equivalent.
    - goldenkey 4y ago
      This is exactly what I meant. Thank you for providing a much more accurate and elegant description.
  - limaoscarjuliet 4y ago
    High cost of suffering (e.g. being ran over by a car when street crossed on red light) and relatively small reward for corresponding success (meh, just crossed the street in the example) might explain it.
- spot5010 4y ago
  This just points to a different way of defining the reward function, no? You would weigh suffering with a high negative weight.
  - anon_123g987 4y ago
    In the simpler case you are optimizing on a single scale between maximum suffering and maximum happiness. In the proposed, more complex case you have two independent scales, the suffering scale (on which you want to primarily minimize), and the happiness scale (on which you want to secondarily maximize).
- anon_123g987 4y ago
  "Negative utilitarianism is a form of negative consequentialism that can be described as the view that people should minimize the total amount of aggregate suffering, or that they should minimize suffering and then, secondarily, maximize the total amount of happiness. It can be considered as a version of utilitarianism that gives greater priority to reducing suffering (negative utility or 'disutility') than to increasing pleasure (positive utility). This differs from classical utilitarianism, which does not claim that reducing suffering is intrinsically more important than increasing happiness." https://en.wikipedia.org/wiki/Negative_utilitarianism https://en.wikipedia.org/wiki/Negative_utilitarianism
- dimatura 4y ago
  In psychology/behavioral economics this is known as loss aversion. Most people would rather avoid losing $5 than not earning $5. But given the rather broad view of reinforcement learning this paper adopts, I would guess that the authors would not view this difference in sign as important. (And indeed, in the more ML/AI view of RL, viewing reward as a negative loss or cost is quite standard and for the most part the sign is seen as an arbitrary convention).
- dqpb 4y ago
  In RL, reward does not mean positive only. You could train an agent on purely negative rewards and it would optimize for the policy that is least negative. Also, +1 and -1 do cancel each other out by definition. If your model has suffering greater than succeeding, then you would assign suffering a greater penalty, like -10 vs +1.
xamuel 4y ago
"Reward Is Enough" is an interesting paper, less for its content, more for its place in the zeitgeist or something. It has very little in the way of novel "theorem proving" but it has sparked quite a lot of discussion. It's a paper which manages to be really great in some sort of subtle way, just because it inspires so much conversation. I'll plug my own response, "Can reinforcement learning learn itself? A reply to 'Reward is enough'", in which I ask: ok, if creating good enough RL agents can lead to AGI, and if AGI can do anything humans can, and if humans can write good RL agents, doesn't that seem to imply it should be possible to use an RL environment to incentivize RL agents to write good RL agents? https://philpapers.org/archive/ALECRL.pdf https://philpapers.org/archive/ALECRL.pdf
- eli_gottlieb 4y ago
  >if creating good enough RL agents can lead to AGI, and if AGI can do anything humans can, and if humans can write good RL agents, doesn't that seem to imply it should be possible to use an RL environment to incentivize RL agents to write good RL agents? Juergen Schmidhuber and Eliezer Yudkowsky are suddenly knocking on your door.
- lern_too_spel 4y ago
  Of course it can, just like a Turing machine can simulate itself.

21 points by optimalsolver 4y ago

Science Direct would think that given that all their reviewers and authors are paid zero dollars but Science Direct turns a profit.

Thorrez 4y ago
If you define reward as dollars, then the reviewers and authors receive no reward.

goldenkey 4y ago

I for one, would greatly forgo reward if it were to avoid suffering. I don't think intelligence is simplistic enough to be driven by just a scalar field. I think like physics, we'll likely need two scalars, or a vector, or a spinor, etc. If "-1" is suffering, and "+1" is reward, then they cancel out. In my own experience, suffering and reward do not cancel out. They have to be on different dimensions. What are your thoughts?

XorNot 4y ago
In studies it seems to take about 5 positive experiences to undo the mental impact of 1 negative experience. [1] So yes - generally these are not considered equal. From almost every perspective, this also makes sense - trading 1 for 1 doesn't lead to you improving your situation. [1] https://www.seekingbalance.com.au/wp-content/uploads/2016/06/BadStrongerThanGood.pdf https://www.seekingbalance.com.au/wp-content/uploads/2016/06...
- anon_123g987 4y ago
  The claim here is even stronger, and somewhat different: that suffering and reward are fundamentally different kind of things, and cannot simply undo each other, not at any exchange rate. The difference is qualitative, not quantitative. For example, if you lose your eyesight, it's such a big loss, that most likely nothing can balance it out ever. It has no "five times as good" equivalent.
  - goldenkey 4y ago
    This is exactly what I meant. Thank you for providing a much more accurate and elegant description.
- limaoscarjuliet 4y ago
  High cost of suffering (e.g. being ran over by a car when street crossed on red light) and relatively small reward for corresponding success (meh, just crossed the street in the example) might explain it.
spot5010 4y ago
This just points to a different way of defining the reward function, no? You would weigh suffering with a high negative weight.
- anon_123g987 4y ago
  In the simpler case you are optimizing on a single scale between maximum suffering and maximum happiness. In the proposed, more complex case you have two independent scales, the suffering scale (on which you want to primarily minimize), and the happiness scale (on which you want to secondarily maximize).
anon_123g987 4y ago
"Negative utilitarianism is a form of negative consequentialism that can be described as the view that people should minimize the total amount of aggregate suffering, or that they should minimize suffering and then, secondarily, maximize the total amount of happiness. It can be considered as a version of utilitarianism that gives greater priority to reducing suffering (negative utility or 'disutility') than to increasing pleasure (positive utility). This differs from classical utilitarianism, which does not claim that reducing suffering is intrinsically more important than increasing happiness." https://en.wikipedia.org/wiki/Negative_utilitarianism https://en.wikipedia.org/wiki/Negative_utilitarianism
dimatura 4y ago
In psychology/behavioral economics this is known as loss aversion. Most people would rather avoid losing $5 than not earning $5. But given the rather broad view of reinforcement learning this paper adopts, I would guess that the authors would not view this difference in sign as important. (And indeed, in the more ML/AI view of RL, viewing reward as a negative loss or cost is quite standard and for the most part the sign is seen as an arbitrary convention).
dqpb 4y ago
In RL, reward does not mean positive only. You could train an agent on purely negative rewards and it would optimize for the policy that is least negative. Also, +1 and -1 do cancel each other out by definition. If your model has suffering greater than succeeding, then you would assign suffering a greater penalty, like -10 vs +1.

xamuel 4y ago

"Reward Is Enough" is an interesting paper, less for its content, more for its place in the zeitgeist or something. It has very little in the way of novel "theorem proving" but it has sparked quite a lot of discussion. It's a paper which manages to be really great in some sort of subtle way, just because it inspires so much conversation. I'll plug my own response, "Can reinforcement learning learn itself? A reply to 'Reward is enough'", in which I ask: ok, if creating good enough RL agents can lead to AGI, and if AGI can do anything humans can, and if humans can write good RL agents, doesn't that seem to imply it should be possible to use an RL environment to incentivize RL agents to write good RL agents? https://philpapers.org/archive/ALECRL.pdf https://philpapers.org/archive/ALECRL.pdf

eli_gottlieb 4y ago
>if creating good enough RL agents can lead to AGI, and if AGI can do anything humans can, and if humans can write good RL agents, doesn't that seem to imply it should be possible to use an RL environment to incentivize RL agents to write good RL agents? Juergen Schmidhuber and Eliezer Yudkowsky are suddenly knocking on your door.
lern_too_spel 4y ago
Of course it can, just like a Turing machine can simulate itself.