7 ms·

Seems like a nothing story. Just looking at the game, there's obviously a constant decision to be made of chase more sheep or instantly die. It sounds like in t

Seems like a nothing story. Just looking at the game, there's obviously a constant decision to be made of chase more sheep or instantly die. It sounds like in the original model they had a max of 20 seconds, so it's not surprising that you would just tank your losses to maximize your score every now and then. Anyone who tries to devise optimal strategies for things should be able to see this isn't especially interesting. Social metaphors are wildly out of place. They say "unintended consequences of a blackbox" but I doubt that's true. Make it a deterministic turn based game and run it through a perfectly transparent optimization model and I wouldn't be surprised to learn this was just the best strategy for the rules they devised. I really hate when people describe an ai as something that cannot be understood because they personally don't understand it.

fny 5y ago
If I remember correctly there were similar scenarios that would occur using that popular Berkeley Pacman universe where he would run into a ghost to avoid the penalty of living for too long.
- hotwire 5y ago
  It reminds me of the thread about the Quake 3 bots, who left alone for several years, figured out that the best approach was to not kill each other. https://i.imgur.com/dx7sVXj.jpg https://i.imgur.com/dx7sVXj.jpg
  - spywaregorilla 5y ago
    Without knowledge of their reward function its difficult to tell if they're converged on this strategy or if its just broken.
- vladTheInhaler 5y ago
  The example you're thinking of is actually in gridworld [1]. As you allud to, one of the parameters of the model is the cost of simply being alive for an additional time-step. If the cost is negative (a reward), then the agent will just sit there forever and accumulate infinite points. If it is zero, it might still just sit there to avoid falling into the hole, which has a large penalty and ends the simulation. As you turn up the dial on the cost of living, the agent starts using more and more aggressive strategies to reach the goal quickly. But if you make it too big, it will just jump in the hole. [1] https://inst.eecs.berkeley.edu/~cs188/fa18/assets/slides/lec9/FA18_cs188_lecture9_MDPs_II_2pp.pdf https://inst.eecs.berkeley.edu/~cs188/fa18/assets/slides/lec...
markwkw 5y ago
Exactly, from technical perspective it's a nothing story. It's interesting, though, how strong of a reaction general public had to this. The story must have strongly resonated with what some folks were already feeling. When you squint (pretend to understand the technology not at all) it's a tragic story. The situation of the wolf seems similar to the situation of some people. Chasing their careers in a highly structured, sort of dehumanized, environment of constant pursuit. "Supreme Intelligence" (that's what a layperson may think of AI) looks at a situation of the wolf and decides that it makes no sense to continue the pursuit. Moreover, what is "optimal" is the most tragic result - suicide.
- wombatmobile 5y ago
  > The story must have strongly resonated with what some folks were already feeling. Yes, because we don't see things as they are, we see them as we are.
  - edoceo 5y ago
    At a Grateful Dead show in Oakland this geezer said to me: Your perception IS your reality man!
    - Darvokis 5y ago
      Schopenhauer: World as representation.
    - BoxOfRain 5y ago
      I'd have loved to have been arond to a Dead show! I know it sounds a little ungrateful coming from someone who lives in a period of unprecedented access to all kinds of wonderful music being written all the time, but there's something about the Dead that really connects with me that I can't quite put my finger on.
      - sitkack 5y ago
        Dark Star Orchestra is your current best bet https://www.youtube.com/watch?v=y8_THRZLSi4 https://www.youtube.com/watch?v=y8_THRZLSi4
rob74 5y ago
It's not surprising from the perspective of an "AI actor". But if you call it a "wolf", most people will assume that it will behave at least roughly like a real-world creature, and the self-preservation instinct is one of the most basic traits of all living beings, so the "AI wolf" not having that is indeed surprising for a layperson.
vnorilo 5y ago
If we play the analogy further: life is suffering, apart from the brief ecstasy of eating sheep. The AI was trying not to suffer, thus chose the boulder. Did my best to translate the (misguided) fitness function to fiction.
- SubiculumCode 5y ago
  Man is born crying, and when he's cried enough, he dies. -Kyoami in Ran. Cutting one's losses early may appear to be the most rational act if trying to minimize an agent's total suffering.
  - 988747 5y ago
    Which is why some forms of Buddhism are basically a cult of death: https://en.wikipedia.org/wiki/Sokushinbutsu https://en.wikipedia.org/wiki/Sokushinbutsu
    - spywaregorilla 5y ago
      > In the video game The Legend of Zelda: Breath of the Wild, the monks in the Ancient Shrines seem to be based on sokushinbutsu. Factoid of the day for sure
  - syntheticnature 5y ago
    David Benatar reached a similar philosophic conclusion due to his utilitarian views, which was amusingly put (with a sort of AI present, no less) in this webcomic: https://existentialcomics.com/comic/253 https://existentialcomics.com/comic/253
    - SubiculumCode 5y ago
      Thanks. I think I just found a new comic to read.
phkahler 5y ago
It's good because most people can understand it. I'd say it's a perfect strategy for a game, but if they're using evolutionary algorithms they should require some form of reproduction for the wolves to carry on. That would make the suicide strategy fail to propagate well. I can also see a number of possible strange outcomes even then.
- jonnycomputer 5y ago
  What mechanism are you thinking of? One in which having offspring is rewarding and so enters into the same learning algorithm, or one in which the learning algorithm/action selection is evolved and differentially conserved?
- spywaregorilla 5y ago
  You're conflating the evolution of the strategy with the idea of the evolution of the actor being controlled by the agent. To give an obvious example, if dying gave 100 points instead of subtracting 10, even the dumbest evolutionary algo would learn to commit suicide asap. The survival of the actor has no intrinsic relevance to how the evolution develops.
  - phkahler 5y ago
    >> You're conflating the evolution of the strategy with the idea of the evolution of the actor being controlled by the agent. Yes. The survival of the actor has no intrinsic relevance to how the evolution develops. No, not in this case. That was my point. That's why the outcome should not be surprising.
mcguire 5y ago
"I really hate when people describe an ai as something that cannot be understood because they personally don't understand it." On the other hand, keep in mind that a significant weakness of most modern AI research is that it's extremely difficult to understand: you have the input, the output, and a bag of statistical weights. In the story, you know the (trivially bad) function that is being optimized; in general you may not. It's not without implications for other systems. Further, "At the end of the day, student and teacher concluded two things: "* The initial bizarre wolf behavior was simply the result of ‘absolute and unfeeling rationality’ exhibited by AI systems. "* It’s hard to predict what conditions matter and what doesn’t to a neural network."
- spywaregorilla 5y ago
  The tooling for understanding complex models is a lot better than what most people assume. > The initial bizarre wolf behavior was simply the result of ‘absolute and unfeeling rationality’ exhibited by AI systems. This is a bad quote. They should not say this. It's a poorly trained agent doing a decent job of a poorly defined environment. Absolute rationality conjures images of some greater thinking but its actually a really stupid model that hit a local maxima. Calling it unfeeling implies the model has some concept of "wolf" and "suicide" but it does not. Replace the visuals with single pixel dots if you want an honest depiction of the room for feelings. > It’s hard to predict what conditions matter and what doesn’t to a neural network." This is generally true, but it isn't true here.

fny 5y ago

If I remember correctly there were similar scenarios that would occur using that popular Berkeley Pacman universe where he would run into a ghost to avoid the penalty of living for too long.

hotwire 5y ago
It reminds me of the thread about the Quake 3 bots, who left alone for several years, figured out that the best approach was to not kill each other. https://i.imgur.com/dx7sVXj.jpg https://i.imgur.com/dx7sVXj.jpg
- spywaregorilla 5y ago
  Without knowledge of their reward function its difficult to tell if they're converged on this strategy or if its just broken.
vladTheInhaler 5y ago
The example you're thinking of is actually in gridworld [1]. As you allud to, one of the parameters of the model is the cost of simply being alive for an additional time-step. If the cost is negative (a reward), then the agent will just sit there forever and accumulate infinite points. If it is zero, it might still just sit there to avoid falling into the hole, which has a large penalty and ends the simulation. As you turn up the dial on the cost of living, the agent starts using more and more aggressive strategies to reach the goal quickly. But if you make it too big, it will just jump in the hole. [1] https://inst.eecs.berkeley.edu/~cs188/fa18/assets/slides/lec9/FA18_cs188_lecture9_MDPs_II_2pp.pdf https://inst.eecs.berkeley.edu/~cs188/fa18/assets/slides/lec...

markwkw 5y ago

Exactly, from technical perspective it's a nothing story. It's interesting, though, how strong of a reaction general public had to this. The story must have strongly resonated with what some folks were already feeling. When you squint (pretend to understand the technology not at all) it's a tragic story. The situation of the wolf seems similar to the situation of some people. Chasing their careers in a highly structured, sort of dehumanized, environment of constant pursuit. "Supreme Intelligence" (that's what a layperson may think of AI) looks at a situation of the wolf and decides that it makes no sense to continue the pursuit. Moreover, what is "optimal" is the most tragic result - suicide.

wombatmobile 5y ago
> The story must have strongly resonated with what some folks were already feeling. Yes, because we don't see things as they are, we see them as we are.
- edoceo 5y ago
  At a Grateful Dead show in Oakland this geezer said to me: Your perception IS your reality man!
  - Darvokis 5y ago
    Schopenhauer: World as representation.
  - BoxOfRain 5y ago
    I'd have loved to have been arond to a Dead show! I know it sounds a little ungrateful coming from someone who lives in a period of unprecedented access to all kinds of wonderful music being written all the time, but there's something about the Dead that really connects with me that I can't quite put my finger on.
    - sitkack 5y ago
      Dark Star Orchestra is your current best bet https://www.youtube.com/watch?v=y8_THRZLSi4 https://www.youtube.com/watch?v=y8_THRZLSi4

rob74 5y ago

It's not surprising from the perspective of an "AI actor". But if you call it a "wolf", most people will assume that it will behave at least roughly like a real-world creature, and the self-preservation instinct is one of the most basic traits of all living beings, so the "AI wolf" not having that is indeed surprising for a layperson.

vnorilo 5y ago

If we play the analogy further: life is suffering, apart from the brief ecstasy of eating sheep. The AI was trying not to suffer, thus chose the boulder. Did my best to translate the (misguided) fitness function to fiction.

SubiculumCode 5y ago
Man is born crying, and when he's cried enough, he dies. -Kyoami in Ran. Cutting one's losses early may appear to be the most rational act if trying to minimize an agent's total suffering.
- 988747 5y ago
  Which is why some forms of Buddhism are basically a cult of death: https://en.wikipedia.org/wiki/Sokushinbutsu https://en.wikipedia.org/wiki/Sokushinbutsu
  - spywaregorilla 5y ago
    > In the video game The Legend of Zelda: Breath of the Wild, the monks in the Ancient Shrines seem to be based on sokushinbutsu. Factoid of the day for sure
- syntheticnature 5y ago
  David Benatar reached a similar philosophic conclusion due to his utilitarian views, which was amusingly put (with a sort of AI present, no less) in this webcomic: https://existentialcomics.com/comic/253 https://existentialcomics.com/comic/253
  - SubiculumCode 5y ago
    Thanks. I think I just found a new comic to read.

phkahler 5y ago

It's good because most people can understand it. I'd say it's a perfect strategy for a game, but if they're using evolutionary algorithms they should require some form of reproduction for the wolves to carry on. That would make the suicide strategy fail to propagate well. I can also see a number of possible strange outcomes even then.

jonnycomputer 5y ago
What mechanism are you thinking of? One in which having offspring is rewarding and so enters into the same learning algorithm, or one in which the learning algorithm/action selection is evolved and differentially conserved?
spywaregorilla 5y ago
You're conflating the evolution of the strategy with the idea of the evolution of the actor being controlled by the agent. To give an obvious example, if dying gave 100 points instead of subtracting 10, even the dumbest evolutionary algo would learn to commit suicide asap. The survival of the actor has no intrinsic relevance to how the evolution develops.
- phkahler 5y ago
  >> You're conflating the evolution of the strategy with the idea of the evolution of the actor being controlled by the agent. Yes. The survival of the actor has no intrinsic relevance to how the evolution develops. No, not in this case. That was my point. That's why the outcome should not be surprising.

mcguire 5y ago

"I really hate when people describe an ai as something that cannot be understood because they personally don't understand it." On the other hand, keep in mind that a significant weakness of most modern AI research is that it's extremely difficult to understand: you have the input, the output, and a bag of statistical weights. In the story, you know the (trivially bad) function that is being optimized; in general you may not. It's not without implications for other systems. Further, "At the end of the day, student and teacher concluded two things: "* The initial bizarre wolf behavior was simply the result of ‘absolute and unfeeling rationality’ exhibited by AI systems. "* It’s hard to predict what conditions matter and what doesn’t to a neural network."

spywaregorilla 5y ago
The tooling for understanding complex models is a lot better than what most people assume. > The initial bizarre wolf behavior was simply the result of ‘absolute and unfeeling rationality’ exhibited by AI systems. This is a bad quote. They should not say this. It's a poorly trained agent doing a decent job of a poorly defined environment. Absolute rationality conjures images of some greater thinking but its actually a really stupid model that hit a local maxima. Calling it unfeeling implies the model has some concept of "wolf" and "suicide" but it does not. Replace the visuals with single pixel dots if you want an honest depiction of the room for feelings. > It’s hard to predict what conditions matter and what doesn’t to a neural network." This is generally true, but it isn't true here.