5 ms·

Greedy AI agents learn to cooperate

91 points by ystad 5y ago

Zenst 5y ago
Interesting and initial thoughts of a hive-mind type approach do seem to be the net result in which the sum of the collective is enabled to perform tasks no individual could. One further upside of this approach would in effect allow and open up a way for different AI systems to integrate and collaborate
ruined 5y ago
this is it. the singularity. ai has finally surpassed human intelligence
- eevilspock 5y ago
  i see what you did there.
- wongarsu 5y ago
  There are plenty of prime examples of greedy humans cooperating. Corruption, cartels, price fixing, collusion in general, ... More seriously: one of the major struggles of society is setting up incentive structures that align the interests of the individual with that of the group, so even people acting in their own self interest still contribute to the greater whole. We only managed to go to the moon because we managed to turn an utterly selfish ideological struggle into a space race.
  - k__ 5y ago
    At the moment it seems that the best performing countries simply found a way to outsource most of their corruption to other countries. I don't know how long this will work...
  - anigbrowl 5y ago
    Those are all rational actors. Many people are irrationally greedy.
    - nickik 5y ago
      Nobody is perfectly rational, and even if they were, because incomplete information would still not result in rational action. This applies for cooperation as well. As soon as you have such an organization the people inside that have the same intensive problem. This equally results in the cooperation itself not being rational. What forms of cooperation works depends again on how to align interest of the people inside them.
- perl4ever 5y ago
  AI could "surpass" human intelligence without ever achieving it. Like imagine a self-replicating solar energy collector that outcompetes every natural plant. It wouldn't replace the ecosystem in a holistic sense, just a Darwinian one. There's no reason dinosaurs couldn't have written better poetry than any creature since.
  - beaconstudios 5y ago
    Computers have long since surpassed human intelligence - there's no way I could perform as many FLOPS in my head as even a really old computer!
    - perl4ever 5y ago
      Not in the sense that I meant. They haven't eradicated all the niches for human intelligence - as a programmer you might overlook the people still making spreadsheets by hand, but I don't think they are endangered, yet.
    - drdeca 5y ago
      A vm inside a vm inside a vm, can’t perform as many operations as the base level machine. It is not clear that such an old machine is capable of doing “more computation” than your brain does. I very much doubt that either is capable of emulating the other in real time.
stevefan1999 5y ago
excuse me if I'm wrong, but this sounds like Ant Colony Optimization to me for some reason?
lettergram 5y ago
I didn't use deep learning, but even markov chains can lead to behavior that looks this way. I managed to create "cooperating" agents via a very very simple model: https://austingwalters.com/modeling-and-building-robotic-sea-slug/ https://austingwalters.com/modeling-and-building-robotic-sea... Effectively, we could show adding this circuit would always be dominate in our simulation. Effectively showing that a single behavior (aggression) could lead to cooperation. I.e. defending ones territory, leads to a cooperation. Genetically this is what our simulations always lead to. The more aggressive other "species" were the quicker and more "robust" the cooperation. Then of course we translated this "learned" behavior into a robot because that seemed like fun lol
- shoo 5y ago
  > even markov chains can lead to behavior that looks this way. What a fun project, thank you for sharing it! re: > Once we created the creatures, we set up their odors, which enable our creatures to smell them. When we have odors and creatures all we then have to do is build a the control components This sounds similar to the "collaborative diffusion" approach to pathfinding [1]. > However, it is possible to view the code on the github page. The link to https://github.com/lettergram/pleuro https://github.com/lettergram/pleuro does not work. Do you still have a copy of the code somewhere? I'm curious to learn more about how you modelled the control loops with Markov chains. Was the rough idea that there are states "forage", "eat", "protect", and then probability of transitions between states depends upon the simulated creature's current state & sensor information about the environment? edit: aha, i can see there's a 2018 publication by Gillette et al "Implementing Goal-Directed Foraging Decisions of a Simpler Nervous System in Simulation" [2] that has corresponding open-sourced cyberslug code [3] [1] Repenning 2006 "Collaborative diffusion: programming antiobjects" https://home.cs.colorado.edu/~ralex/papers/PDF/OOPSLA06antiobjects.pdf https://home.cs.colorado.edu/~ralex/papers/PDF/OOPSLA06antio... [2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5830682/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5830682/ [3] https://github.com/derekca/cyberslug https://github.com/derekca/cyberslug not the same codebase, but same advisor, so the code may be some loose relative. alas, the linked codebase seems rather "academic research quality", in the sense that there there's 5000 lines of netlogo cyberslug code and there don't appear to be any automated tests.
AndrewKemendo 5y ago
It seems like there might be a trend to use evolutionary learning (genetic algorithms) in conjunction with RL. That is what Deepmind did with the recent open-ended play agents [1] in order to create more generally capable agents (among a lot of other specific things). In the DeepMind case the largest variable is the environment. In the Intel case the largest variable is the the humanoid component. However both are using task specificity in some narrow task to then bootstrap a combination function to solve more general tasks. That's definitely a good approach and definitely has a lot of good history showing it's effectiveness. The reason it's notable is because of how it's acknowledging that ensemble methods really are powerful and (especially with RL) that those are likely the way toward extremely general and flexible agents. [1]https://deepmind.com/blog/article/generally-capable-agents-emerge-from-open-ended-play https://deepmind.com/blog/article/generally-capable-agents-e...
hoseja 5y ago
If I, your friendly neighborhood software engineer wanted to do reinforcement training for a simple board game, where would I even start?
- svalorzen 5y ago
  I maintain a repository of many implementations of classical (tabular) RL algorithms [1] which you might enjoy playing with when starting out. I use it for both research and for student projects. The advantage of avoiding NNs when starting out is that it is much simpler to inspect the inner workings of an algorithm to see whether it's working or not. I'm always happy to help if something is unclear or difficult so feel free to open issues there :) [1]: https://github.com/Svalorzen/AI-Toolbox https://github.com/Svalorzen/AI-Toolbox
- TylerLives 5y ago
  Kaggle has competitions for games and many people use RL there. Looking through their solutions might be of use to you.
- lostmsu 5y ago
  Alpha Zero would be a good start (a generalized version of Alpha Go, that started the current AI hype cycle) https://duckduckgo.com/?q=alpha+zero+github&t=ffab&ia=web https://duckduckgo.com/?q=alpha+zero+github&t=ffab&ia=web

91 points by ystad 5y ago

Interesting and initial thoughts of a hive-mind type approach do seem to be the net result in which the sum of the collective is enabled to perform tasks no individual could. One further upside of this approach would in effect allow and open up a way for different AI systems to integrate and collaborate

ruined 5y ago

this is it. the singularity. ai has finally surpassed human intelligence

eevilspock 5y ago
i see what you did there.
wongarsu 5y ago
There are plenty of prime examples of greedy humans cooperating. Corruption, cartels, price fixing, collusion in general, ... More seriously: one of the major struggles of society is setting up incentive structures that align the interests of the individual with that of the group, so even people acting in their own self interest still contribute to the greater whole. We only managed to go to the moon because we managed to turn an utterly selfish ideological struggle into a space race.
- k__ 5y ago
  At the moment it seems that the best performing countries simply found a way to outsource most of their corruption to other countries. I don't know how long this will work...
- anigbrowl 5y ago
  Those are all rational actors. Many people are irrationally greedy.
  - nickik 5y ago
    Nobody is perfectly rational, and even if they were, because incomplete information would still not result in rational action. This applies for cooperation as well. As soon as you have such an organization the people inside that have the same intensive problem. This equally results in the cooperation itself not being rational. What forms of cooperation works depends again on how to align interest of the people inside them.
perl4ever 5y ago
AI could "surpass" human intelligence without ever achieving it. Like imagine a self-replicating solar energy collector that outcompetes every natural plant. It wouldn't replace the ecosystem in a holistic sense, just a Darwinian one. There's no reason dinosaurs couldn't have written better poetry than any creature since.
- beaconstudios 5y ago
  Computers have long since surpassed human intelligence - there's no way I could perform as many FLOPS in my head as even a really old computer!
  - perl4ever 5y ago
    Not in the sense that I meant. They haven't eradicated all the niches for human intelligence - as a programmer you might overlook the people still making spreadsheets by hand, but I don't think they are endangered, yet.
  - drdeca 5y ago
    A vm inside a vm inside a vm, can’t perform as many operations as the base level machine. It is not clear that such an old machine is capable of doing “more computation” than your brain does. I very much doubt that either is capable of emulating the other in real time.

stevefan1999 5y ago

excuse me if I'm wrong, but this sounds like Ant Colony Optimization to me for some reason?

lettergram 5y ago

I didn't use deep learning, but even markov chains can lead to behavior that looks this way. I managed to create "cooperating" agents via a very very simple model: https://austingwalters.com/modeling-and-building-robotic-sea-slug/ https://austingwalters.com/modeling-and-building-robotic-sea... Effectively, we could show adding this circuit would always be dominate in our simulation. Effectively showing that a single behavior (aggression) could lead to cooperation. I.e. defending ones territory, leads to a cooperation. Genetically this is what our simulations always lead to. The more aggressive other "species" were the quicker and more "robust" the cooperation. Then of course we translated this "learned" behavior into a robot because that seemed like fun lol

shoo 5y ago
> even markov chains can lead to behavior that looks this way. What a fun project, thank you for sharing it! re: > Once we created the creatures, we set up their odors, which enable our creatures to smell them. When we have odors and creatures all we then have to do is build a the control components This sounds similar to the "collaborative diffusion" approach to pathfinding [1]. > However, it is possible to view the code on the github page. The link to https://github.com/lettergram/pleuro https://github.com/lettergram/pleuro does not work. Do you still have a copy of the code somewhere? I'm curious to learn more about how you modelled the control loops with Markov chains. Was the rough idea that there are states "forage", "eat", "protect", and then probability of transitions between states depends upon the simulated creature's current state & sensor information about the environment? edit: aha, i can see there's a 2018 publication by Gillette et al "Implementing Goal-Directed Foraging Decisions of a Simpler Nervous System in Simulation" [2] that has corresponding open-sourced cyberslug code [3] [1] Repenning 2006 "Collaborative diffusion: programming antiobjects" https://home.cs.colorado.edu/~ralex/papers/PDF/OOPSLA06antiobjects.pdf https://home.cs.colorado.edu/~ralex/papers/PDF/OOPSLA06antio... [2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5830682/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5830682/ [3] https://github.com/derekca/cyberslug https://github.com/derekca/cyberslug not the same codebase, but same advisor, so the code may be some loose relative. alas, the linked codebase seems rather "academic research quality", in the sense that there there's 5000 lines of netlogo cyberslug code and there don't appear to be any automated tests.

AndrewKemendo 5y ago

It seems like there might be a trend to use evolutionary learning (genetic algorithms) in conjunction with RL. That is what Deepmind did with the recent open-ended play agents [1] in order to create more generally capable agents (among a lot of other specific things). In the DeepMind case the largest variable is the environment. In the Intel case the largest variable is the the humanoid component. However both are using task specificity in some narrow task to then bootstrap a combination function to solve more general tasks. That's definitely a good approach and definitely has a lot of good history showing it's effectiveness. The reason it's notable is because of how it's acknowledging that ensemble methods really are powerful and (especially with RL) that those are likely the way toward extremely general and flexible agents. [1]https://deepmind.com/blog/article/generally-capable-agents-emerge-from-open-ended-play https://deepmind.com/blog/article/generally-capable-agents-e...

hoseja 5y ago

If I, your friendly neighborhood software engineer wanted to do reinforcement training for a simple board game, where would I even start?

svalorzen 5y ago
I maintain a repository of many implementations of classical (tabular) RL algorithms [1] which you might enjoy playing with when starting out. I use it for both research and for student projects. The advantage of avoiding NNs when starting out is that it is much simpler to inspect the inner workings of an algorithm to see whether it's working or not. I'm always happy to help if something is unclear or difficult so feel free to open issues there :) [1]: https://github.com/Svalorzen/AI-Toolbox https://github.com/Svalorzen/AI-Toolbox
TylerLives 5y ago
Kaggle has competitions for games and many people use RL there. Looking through their solutions might be of use to you.
lostmsu 5y ago
Alpha Zero would be a good start (a generalized version of Alpha Go, that started the current AI hype cycle) https://duckduckgo.com/?q=alpha+zero+github&t=ffab&ia=web https://duckduckgo.com/?q=alpha+zero+github&t=ffab&ia=web