6 ms·

Capture the Flag: the emergence of complex cooperative agents

73 points by yigitdemirag 7y ago

DuskStar 7y ago
This feels a lot like OpenAI 5, but with a few key differences - for one, it seems to handle randomly generated maps and other unexpected situations, and each agent does not have full knowledge of the others on its team. Really cool! And it would be hilarious if this ends up giving us video game enemies that don't suck. (Until, of course, someone accidentally creates an RTS AI that manages to 'break out' and conquer the world or something)
- pygy_ 7y ago
  I wonder who from Google, Facebook and OpenAI will be the first to productize a "CEO as a service" AI.
  - wnkrshm 7y ago
    I think it's an interesting question when this will start to show up - not in terms of managing everything including taxes but in terms of making automated business decisions. In niches where business decisions are data driven, connecting a supply in some market to a demand in another, the choice of what to buy and sell could conceivably be automated. Like gauging demand for mug/thermos/branded item du jour on amazon, checking whether similar items exist on alibaba and then setting up an amazon market place that resells your slightly rebranded but identical item from alibaba. Of course this all hinges on the data - an algorithm is prone to fall into traps that some humans fall into (i.e. the company you buy from only sells the box and not the product).
- stefs 7y ago
  evolving agents like those are very good at finding bugs to abuse. so, if they'd find a zero day to break out of the quake-sandbox, what would they do? i guess, for the longest time they'd just use the DMA into the sandbox to increase their score (i.e. cheating by developing and using external tools), as their fitness function still weights CTF wins only. to (accidentally) change their fitness function to change it to "conquer the world" they'd need to modify a lot of different processes at once. IMO the chance is negligible for now.
  - DuskStar 7y ago
    I think the (very sci-fy) fear there isn't "we gave this agent a goal of 'score xyz points' and it escaped the sandbox to increment the points counter" but instead "we gave this agent a goal of 'conquer the world' and made it think that the game it was in was the world, and then it discovered otherwise"
kuu 7y ago
Super interesting!
7373737373 7y ago
Do these agents evolve to communicate with each other (e.g. by wiggling)? With OpenAI5, it would have been cool to restrict inter-agent communication to an audio channel. Would a discrete language evolve?
- Yajirobe 7y ago
  That is a very cool question. Now I am interested to see if anything like that has been done.
- place4you 7y ago
  I would love to see that, maybe a bird-like communication would occur.
- pulkitsh1234 7y ago
  'language' as we know is quite ambiguous and an inefficient way of communication. my guess is that what they will probably evolve is some kind of an audio encoding format for the exact information they want to convey to the other entity, like literally sending it control inputs over audio.
  - M2Ys4U 7y ago
    Given that the AIs were playing with humans, perhaps using speech to text/text to speech as an interface in order to be "human compatible" would overcome some of that.
  - pgt 7y ago
    What is the difference between what you are describing and a language?
    - londons_explore 7y ago
      Human language has a second or more of latency. Shouting "he's behind you!!", together with the other person hearing it, understanding it, and doing something about is is a very slow process. AI would probably come up with signals which have less latency.
YeGoblynQueenne 7y ago
>> The agents are never told anything about the rules of the game, yet learn about fundamental game concepts and effectively develop an intuition for CTF. Here we go again- throwing around big words, "intuition", and marring an otherwise interesting piece of work. From a cursory glance at the relevant figure, "A look into how our agents represent the game world" - it looks like what is being learned is very much a goood old behaviour tree. For instance, the figure suggests that agents learned to react to situations like "Agent flag at base & opponent flag at base & not respawning & agent in home base". So basically, the condition part of an IF-THEN-ELSE rule. Why is this called an "intuition" rather than a "rule"? From my reading, the only reason is that it was learned by a deep neural net by reinforcement learning without explicit supervision, i.e. without anyone telling the agent "learn this rule". That's a very narrow, procrustean, definition of intuition. Is it really what most people would mean by "intuition"? Is it even close? Who knows- nobody can tell what most people would mean by "intuition". There's a dictionary definition, but chances are most people would not know it by heart. So it's very hard to even say "that's not what intuition means"- one would just be begging the question. So why use such a vague term in an article like this, that is already pretty impressive stuff? What do such meaningless claims add to the result? I don't get it.
- bschne 7y ago
  I'm not up to date on the relevant research, but isn't most learned or innate intuition just a set of fuzzy rules based on observation (be it visual or other sensory perception; or pre-defined behaviour in the case of innate intuition)? If I play tennis for a while, I will get better at predicting where the ball will go after my opponent hits it merely because I have seen many combinations of shots and subsequent ball trajectories. I think it would be weird to not call it an intuition just because it was learned by a deep neural net in this case.
  - YeGoblynQueenne 7y ago
    In my experience, in the context of artificial and human intelligence, any statement of the form "X is just Y" or, equivalently, "Isn't X just Y" is neither sound nor complete and X and Y are not well defined. In your example, for instance- I have no idea what are "fuzzy rules" and how they apply to tennis playing, or how they relate to predicting the position of the ball etc.

73 points by yigitdemirag 7y ago

This feels a lot like OpenAI 5, but with a few key differences - for one, it seems to handle randomly generated maps and other unexpected situations, and each agent does not have full knowledge of the others on its team. Really cool! And it would be hilarious if this ends up giving us video game enemies that don't suck. (Until, of course, someone accidentally creates an RTS AI that manages to 'break out' and conquer the world or something)

pygy_ 7y ago
I wonder who from Google, Facebook and OpenAI will be the first to productize a "CEO as a service" AI.
- wnkrshm 7y ago
  I think it's an interesting question when this will start to show up - not in terms of managing everything including taxes but in terms of making automated business decisions. In niches where business decisions are data driven, connecting a supply in some market to a demand in another, the choice of what to buy and sell could conceivably be automated. Like gauging demand for mug/thermos/branded item du jour on amazon, checking whether similar items exist on alibaba and then setting up an amazon market place that resells your slightly rebranded but identical item from alibaba. Of course this all hinges on the data - an algorithm is prone to fall into traps that some humans fall into (i.e. the company you buy from only sells the box and not the product).
stefs 7y ago
evolving agents like those are very good at finding bugs to abuse. so, if they'd find a zero day to break out of the quake-sandbox, what would they do? i guess, for the longest time they'd just use the DMA into the sandbox to increase their score (i.e. cheating by developing and using external tools), as their fitness function still weights CTF wins only. to (accidentally) change their fitness function to change it to "conquer the world" they'd need to modify a lot of different processes at once. IMO the chance is negligible for now.
- DuskStar 7y ago
  I think the (very sci-fy) fear there isn't "we gave this agent a goal of 'score xyz points' and it escaped the sandbox to increment the points counter" but instead "we gave this agent a goal of 'conquer the world' and made it think that the game it was in was the world, and then it discovered otherwise"

kuu 7y ago

Super interesting!

7373737373 7y ago

Do these agents evolve to communicate with each other (e.g. by wiggling)? With OpenAI5, it would have been cool to restrict inter-agent communication to an audio channel. Would a discrete language evolve?

Yajirobe 7y ago
That is a very cool question. Now I am interested to see if anything like that has been done.
place4you 7y ago
I would love to see that, maybe a bird-like communication would occur.
pulkitsh1234 7y ago
'language' as we know is quite ambiguous and an inefficient way of communication. my guess is that what they will probably evolve is some kind of an audio encoding format for the exact information they want to convey to the other entity, like literally sending it control inputs over audio.
- M2Ys4U 7y ago
  Given that the AIs were playing with humans, perhaps using speech to text/text to speech as an interface in order to be "human compatible" would overcome some of that.
- pgt 7y ago
  What is the difference between what you are describing and a language?
  - londons_explore 7y ago
    Human language has a second or more of latency. Shouting "he's behind you!!", together with the other person hearing it, understanding it, and doing something about is is a very slow process. AI would probably come up with signals which have less latency.

YeGoblynQueenne 7y ago

>> The agents are never told anything about the rules of the game, yet learn about fundamental game concepts and effectively develop an intuition for CTF. Here we go again- throwing around big words, "intuition", and marring an otherwise interesting piece of work. From a cursory glance at the relevant figure, "A look into how our agents represent the game world" - it looks like what is being learned is very much a goood old behaviour tree. For instance, the figure suggests that agents learned to react to situations like "Agent flag at base & opponent flag at base & not respawning & agent in home base". So basically, the condition part of an IF-THEN-ELSE rule. Why is this called an "intuition" rather than a "rule"? From my reading, the only reason is that it was learned by a deep neural net by reinforcement learning without explicit supervision, i.e. without anyone telling the agent "learn this rule". That's a very narrow, procrustean, definition of intuition. Is it really what most people would mean by "intuition"? Is it even close? Who knows- nobody can tell what most people would mean by "intuition". There's a dictionary definition, but chances are most people would not know it by heart. So it's very hard to even say "that's not what intuition means"- one would just be begging the question. So why use such a vague term in an article like this, that is already pretty impressive stuff? What do such meaningless claims add to the result? I don't get it.

bschne 7y ago
I'm not up to date on the relevant research, but isn't most learned or innate intuition just a set of fuzzy rules based on observation (be it visual or other sensory perception; or pre-defined behaviour in the case of innate intuition)? If I play tennis for a while, I will get better at predicting where the ball will go after my opponent hits it merely because I have seen many combinations of shots and subsequent ball trajectories. I think it would be weird to not call it an intuition just because it was learned by a deep neural net in this case.
- YeGoblynQueenne 7y ago
  In my experience, in the context of artificial and human intelligence, any statement of the form "X is just Y" or, equivalently, "Isn't X just Y" is neither sound nor complete and X and Y are not well defined. In your example, for instance- I have no idea what are "fuzzy rules" and how they apply to tennis playing, or how they relate to predicting the position of the ball etc.