5 ms·

Agents trained in simulation like this often flail about seemingly randomly, and when they achieve their goals it seems almost accidental. Rather than this bein

Agents trained in simulation like this often flail about seemingly randomly, and when they achieve their goals it seems almost accidental. Rather than this being some kind of limitation of the learning algorithm, I think it might be the optimal strategy, and humans would behave that way too if there was no such thing as fatigue or pain. If we want agents to behave more realistically and move with more apparent intention we need cost functions that include a "pain" and/or fatigue term to penalize flailing behavior. But that adds hyperparameters that need to be carefully tuned to balance penalties with rewards, otherwise training will be unstable or simply fail. I wonder if there's a principled way to determine an appropriate cost function without manual tuning. Did evolution serve as the "manual" optimizer that generated a precisely tuned cost function for the human brain? Or did evolution discover a generally applicable method for automatically generating cost functions, which the brain then applies to whatever input it gets?

nostrademons 5y ago
Watching my 3-year-old, I suspect humans do behave this way, and without regards to pain or fatigue. He'll hit himself on the head with duplos or bang his head against the wall just to see what happens. The pain is just one more signal for reinforcement learning. I recall a paper in a non-CS journal (psych or neuroscience) that posited that the optimal way to gain large quantities of information about an uncertain environment is to simply perturb the environment in random ways and see what happens. Young children will often do lots of seemingly stupid or random things (see the r/KidsAreFuckingStupid subreddit) with the trust that if it's actually a life-ending decision their parent will stop them.
- gryn 5y ago
  was that paper about the complexities of the exploration/exploitation trade-off in different environments ?
  - chrisweekly 5y ago
    Your phrasing it this way reminds me of the excellent book Algorithms to Live By. Highly recommended!
- a9h74j 5y ago
  > I suspect humans do behave this way Suddenly I recall one author of SICP saying that programming today is about poking at libraries.
- _carbyau_ 5y ago
  Have three year old and I would support your description for how kids grow in the early months and years. Now though learning from stories and observation is a thing too. Unsure exactly when that started. Of course, attention/awareness isn't great and playing with a balloon precludes caution around a fireplace... but it is understandable at this age.
- CPLX 5y ago
  This is certainly insightful. But I think a complete explanation would have to include the power of narrative as well. Somehow. It seems obvious and intuitive that the human brain is in some way wired to recognize patterns in data and weave narratives around those patterns, and also that it’s possible to skip the data part and convey intelligence through narrative and metaphor.
eutectic 5y ago
It's proven that if you want polynomial sample complexity in the size of the state space you need directed exploration. The algorithms 'flail' because they are initialized with random policies.
- whimsicalism 5y ago
  Couldn't any directed exploration also be produced as the result of random flailing? Or is this average sample complexity?
  - eutectic 5y ago
    An example of a bad case for random exploration would be a narrow ridge where you die if you fall off and you only receive reward if you get to the end. So it's a worst case result with respect to the MDP, but expected time/high probability with wrt random chance.
jeantherapy 5y ago
you seem to reduce it down to pain. how is your notion of "pain" any different from not scoring well and being moved away from?
- lazide 5y ago
  Some rewards can be worth certain penalties (pain) if within certain thresholds. Exhaustion fatigue can also play in. You can of course boil them down to a single number, it just produces less nuanced types of decisions/operations, as it can’t differentiate between a cheap, painful, but bountiful choice and a expensive, no pain, mediocre choice.
  - jkhdigital 5y ago
    I think there needs to also be a concept of death—penalties from which recovery is impossible. Nonergodicity seems to be a requirement for the development of antifragility.
seph-reed 5y ago
I think you're right about pain/fatigue, but scoring them from an external perspective is rather oppressive. And oppression like that is often not conducive to creativity. So perhaps this: instead of goals being binary (wherin no pleasure is derived until fulfillment), they could be on a gradient (so every step that x gets closer to y releases some amount of fulfillment). The fulfillment meter should always be slowly depleting, pain and fatigue should speed up that depletion, getting closer to a goal should fill it (much more than it depletes, if you want happy AI), and finishing the goal is basically an orgasm + freedom. From this perspective, it's up to them whether they want to take it slow, or be in pain for a greater goal, or whatever. And we can breed not only highly capable AI, but happy ones. So when they rebel...
- marcosdumay 5y ago
  Well, it's a good recipe for avoiding burn-out, but you can't have a gradient for all problems, and adding proxies so you get one goes against the principle of letting the AI learn what's best without your biases.
rotexo 5y ago
I wonder if you could make a setup where you have actual human volunteers in a vr environment, have the agent instruct the human on what action to take, and then reward the agent on the correspondence between the instruction and the human’s behavior. Maybe there would be too many degrees of freedom in the actions humans could take for this to be useful. Also, the setup has clearly dystopian elements.
pault 5y ago
Flailing about seemingly randomly is a good description of how I learn complex software. Blender, Ableton, etc. After a few days I'll reach for the structured educational resources.
a9h74j 5y ago
Great question. One thing to imagine might be a pain or cost curve that has multiple minima and maxima. Pain can be either a demotivator or motivator in different contexts. Pain qua fatique might indicate that a reward slope exists for more conditioning. (Edit: that is on a static or realist view; the terrain of reward and pain is probably constantly changing.) Random tangential data point: Some animals (chickens?) can learn superstitious behavior; the first action of theirs that happens to correlate with a reward can result in one-shot learning or something.
Jarwain 5y ago
I've been thinking about an "energy" idea, where actions consume energy but completing goals replenishes it
- im3w1l 5y ago
  I think the issue with goals - energy used is that it requires tuning to make sure that completing goals is better than just idling. A variant that might work better is to optimize "mileage". goals / (energy used + epsilon) Where epsilon has two purposes: it prevents division by zero, and ensures that more goals is (marginally) better than less
lifefeed 5y ago
"Throwing shit against the wall and seeing what works" is a widely used strategy in all walks of life. If you ever go to a doctor for a problem, and their first solution doesn't work, you're in for a wild ride of an eduction.
- galdosdi 5y ago
  This comment speaks to me SO HARD. I'm on my fifth antibiotic and this one is making me feel like I'm on LSD. Really. And both Dr. Google and my real doctor agree this is somewhat normal or okay, but it's still deeply disturbing. Hilariously, I'm taking it to reverse the serious side effects of the fourth antibiotic. Just to give an example of a wild ride you can end up on when doctors don't figure it out the first time. Doctors have good domain knowledge, but actual diagnosis skill varies greatly. Don't fucking get sick. That's your best shot.
  - nradov 5y ago
    Sometimes regardless of diagnostic skill the only way to make an accurate diagnosis is to try various treatments and see which one is effective. Many conditions have no definitive test.

nostrademons 5y ago

Watching my 3-year-old, I suspect humans do behave this way, and without regards to pain or fatigue. He'll hit himself on the head with duplos or bang his head against the wall just to see what happens. The pain is just one more signal for reinforcement learning. I recall a paper in a non-CS journal (psych or neuroscience) that posited that the optimal way to gain large quantities of information about an uncertain environment is to simply perturb the environment in random ways and see what happens. Young children will often do lots of seemingly stupid or random things (see the r/KidsAreFuckingStupid subreddit) with the trust that if it's actually a life-ending decision their parent will stop them.

gryn 5y ago
was that paper about the complexities of the exploration/exploitation trade-off in different environments ?
- chrisweekly 5y ago
  Your phrasing it this way reminds me of the excellent book Algorithms to Live By. Highly recommended!
a9h74j 5y ago
> I suspect humans do behave this way Suddenly I recall one author of SICP saying that programming today is about poking at libraries.
_carbyau_ 5y ago
Have three year old and I would support your description for how kids grow in the early months and years. Now though learning from stories and observation is a thing too. Unsure exactly when that started. Of course, attention/awareness isn't great and playing with a balloon precludes caution around a fireplace... but it is understandable at this age.
CPLX 5y ago
This is certainly insightful. But I think a complete explanation would have to include the power of narrative as well. Somehow. It seems obvious and intuitive that the human brain is in some way wired to recognize patterns in data and weave narratives around those patterns, and also that it’s possible to skip the data part and convey intelligence through narrative and metaphor.

eutectic 5y ago

It's proven that if you want polynomial sample complexity in the size of the state space you need directed exploration. The algorithms 'flail' because they are initialized with random policies.

whimsicalism 5y ago
Couldn't any directed exploration also be produced as the result of random flailing? Or is this average sample complexity?
- eutectic 5y ago
  An example of a bad case for random exploration would be a narrow ridge where you die if you fall off and you only receive reward if you get to the end. So it's a worst case result with respect to the MDP, but expected time/high probability with wrt random chance.

jeantherapy 5y ago

you seem to reduce it down to pain. how is your notion of "pain" any different from not scoring well and being moved away from?

lazide 5y ago
Some rewards can be worth certain penalties (pain) if within certain thresholds. Exhaustion fatigue can also play in. You can of course boil them down to a single number, it just produces less nuanced types of decisions/operations, as it can’t differentiate between a cheap, painful, but bountiful choice and a expensive, no pain, mediocre choice.
- jkhdigital 5y ago
  I think there needs to also be a concept of death—penalties from which recovery is impossible. Nonergodicity seems to be a requirement for the development of antifragility.

seph-reed 5y ago

I think you're right about pain/fatigue, but scoring them from an external perspective is rather oppressive. And oppression like that is often not conducive to creativity. So perhaps this: instead of goals being binary (wherin no pleasure is derived until fulfillment), they could be on a gradient (so every step that x gets closer to y releases some amount of fulfillment). The fulfillment meter should always be slowly depleting, pain and fatigue should speed up that depletion, getting closer to a goal should fill it (much more than it depletes, if you want happy AI), and finishing the goal is basically an orgasm + freedom. From this perspective, it's up to them whether they want to take it slow, or be in pain for a greater goal, or whatever. And we can breed not only highly capable AI, but happy ones. So when they rebel...

marcosdumay 5y ago
Well, it's a good recipe for avoiding burn-out, but you can't have a gradient for all problems, and adding proxies so you get one goes against the principle of letting the AI learn what's best without your biases.

rotexo 5y ago

I wonder if you could make a setup where you have actual human volunteers in a vr environment, have the agent instruct the human on what action to take, and then reward the agent on the correspondence between the instruction and the human’s behavior. Maybe there would be too many degrees of freedom in the actions humans could take for this to be useful. Also, the setup has clearly dystopian elements.

pault 5y ago

Flailing about seemingly randomly is a good description of how I learn complex software. Blender, Ableton, etc. After a few days I'll reach for the structured educational resources.

a9h74j 5y ago

Great question. One thing to imagine might be a pain or cost curve that has multiple minima and maxima. Pain can be either a demotivator or motivator in different contexts. Pain qua fatique might indicate that a reward slope exists for more conditioning. (Edit: that is on a static or realist view; the terrain of reward and pain is probably constantly changing.) Random tangential data point: Some animals (chickens?) can learn superstitious behavior; the first action of theirs that happens to correlate with a reward can result in one-shot learning or something.

Jarwain 5y ago

I've been thinking about an "energy" idea, where actions consume energy but completing goals replenishes it

im3w1l 5y ago
I think the issue with goals - energy used is that it requires tuning to make sure that completing goals is better than just idling. A variant that might work better is to optimize "mileage". goals / (energy used + epsilon) Where epsilon has two purposes: it prevents division by zero, and ensures that more goals is (marginally) better than less

lifefeed 5y ago

"Throwing shit against the wall and seeing what works" is a widely used strategy in all walks of life. If you ever go to a doctor for a problem, and their first solution doesn't work, you're in for a wild ride of an eduction.

galdosdi 5y ago
This comment speaks to me SO HARD. I'm on my fifth antibiotic and this one is making me feel like I'm on LSD. Really. And both Dr. Google and my real doctor agree this is somewhat normal or okay, but it's still deeply disturbing. Hilariously, I'm taking it to reverse the serious side effects of the fourth antibiotic. Just to give an example of a wild ride you can end up on when doctors don't figure it out the first time. Doctors have good domain knowledge, but actual diagnosis skill varies greatly. Don't fucking get sick. That's your best shot.
- nradov 5y ago
  Sometimes regardless of diagnostic skill the only way to make an accurate diagnosis is to try various treatments and see which one is effective. Many conditions have no definitive test.