12 ms·

Learning to Cooperate, Compete, and Communicate

199 points by maccaw 9y ago

EGreg 9y ago
Am I the only one that finds this scary? As I have repeatedly said, so far the intelligence we have been producing is NOT the kind that applies abstract logic rules to figure out meaning. It is the kind that takes full advantage of computers' strengths: perfect copying and speed. So good ideas are copied and propagate. Neural networks are just this on steroids... basically extracting signals from noise and doing a search in a space to maximize something, and storing the results. Humans were able to transmit knowledge, then produce books etc. Now bits can be perfectly copied with checksums. This isn't general intelligence in the human sense. But that's what makes it scary. It can solve these problems with brute force. Resistance soon may really be futile. Not just in running, avoiding capture and death. But also in ANY human system we rely on, including voting, due process of law, trust, reputation, family, sex, humor, etc.
- zitterbewegung 9y ago
  As someone that is teaching themselves Machine learning solving problems with "brute force" doesn't actually happen and Neural Networks are a way around using brute force. It seems that you are caught in the hype around machine learning. Systems that can copy ANY human system are very far off. Also, there are human systems where AI's would not be found to be acceptable.
  - EGreg 9y ago
    Just to be clear: I didn't say they can copy any human system. I am saying they can find new ways to beat humans or groups of humans at the games we have set up. Chess and Go are just examples. Trust and Reputation are easily attacked. Voting is easily manipulated (look at gerrymandering and the increasingly shrill "fake news" claims.) Look, they are already better at making diagnoses than doctors. I am saying that the machines won't really "understand" the human meaning behind things. They will just have "deep" insight in having explored a huge amount of variations and extracted the combinations that maximize some goal. You think you are persuasive now? You think comedians are funny? Fast forrward 17 years when any software can deliver blistering putdowns to rappers, destroy the best comedian to the point where every human agrees the guy looks dull in comparison. The AI that can easily outcompete guys at wooing a girl online. And so on. Forget the Turing test. We are in for a future where programs simply "solve" the systems we set up faster and better than we can. And share that knowledge in compact form (such as artificial neural net calibration). Most of our systems rely on the inefficiency of an attacker. That they can't instantly start up 3,000,000 businesses or do sybil attacks in voting or destroy someone's credit overnight or win an election. But that is going to change. And I am saying that's just in the near future. Humor and trust is a little further.
    - cryoshon 9y ago
      yep. put simply: they will be better, once they are correctly configured. the task doesn't matter. there's nothing that's off limits, in the long run. the trick will be getting them to serve us on command, and to get them to hold back when we'd rather do something imperfectly and interpersonally rather than perfectly and computationally.
      - marcosdumay 9y ago
        We'll never be able to have them serve us, if for no other reason, because "we" have many diverse interests that are often antagonist between themselves. The best we can achieve is getting some of them to serve us on command, to protect us from other ones serving other people.
    - deleted 9y ago
      [deleted]
    - readonlydanger 9y ago
      Unfortunately, many of your assertions are inaccurate. As far as game playing goes, the ways in which AI agents beat human opponents is simply a matter of computers being able to process more information in a given period of time. The selection and evaluation criteria are actually pretty similar to human decision-making, if a bit more systematic. I can expand on this point in another comment, if you'd like. Chess and Go are not solved games, and will likely not be for some time (10+ years, for Go at least). Solving non-deterministic problems are even harder, and many likely will not be accomplished within our lifetimes[1]. There is no AI system that can generally make more accurate diagnoses than a physician. We're not there quite yet, and it will be a while. The best we've come are some pretty advanced expert systems, but these require very heavy input from physicians[2]. Please provide sources to validate your claims on the progress of AI. Extrapolation from data is dangerous, extrapolation from falsity is ignorance. As far as your take on understanding goes, the human "meaning" of things is inherently subjective. If we define "meaning" as some thing's value or role based on environmental context, then just like humans, any artificially intelligent system will only be able to determine meaning based on observations of the environment and both individual and collective experience. It's interesting that you seem to be focusing primarily on social domains that have inherently "human" contexts. Beyond misunderstanding the point of this paper, and the way AI systems work, I think you're missing the point. AI is, and for the time being will remain, an extension of the human mind. The decision-making needs to be developed (at some core level) by a human. Those goals need to be set by a human. The experiences and observational capabilities ultimately need to be determined by a human. Even AI systems that build other AI systems need to be directed to do so, and with strict goals, set by a human [3]. I highly suggest you take a moment to read openai's mission statement: https://openai.com/about/ https://openai.com/about/. AI is a tool. And like any powerful tool it must be used responsibly, freely, and openly. Openai is pursuing this goal and making efforts to ensure that this tool is available to as many people as possible to avoid the abuses implicit to your concerns. You obviously have an interest in AI and some knowledge of the field, but I worry your comments veer a bit towards fear-mongering. I suggest you use openai and resources like it to enrich your knowledge of both the advances and concerns of AI, because those are important, and we definitely need people thinking about these things. Ultimately, you are absolutely correct that eventually these systems will probably have the technical capability to influence elections, the economy, and more. But the only way they will is under the direction of humans. It it not the machine you should fear, but the man behind it. The same thing you ought to have been fearing all along. 1. http://fragrieu.free.fr/SearchingForSolutions.pdf http://fragrieu.free.fr/SearchingForSolutions.pdf 2. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1307157/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1307157/ 3. https://arxiv.org/abs/1611.02779 https://arxiv.org/abs/1611.02779
- hacker_9 9y ago
  Still, better this is done in the open. Otherwise Russia and China will just continue in secret, which is definitely a world I don't want to live in.
  - EGreg 9y ago
    I agree it should be done in the open. Doesn't make it much less scary.
    - hacker_9 9y ago
      It does make it less scary because we progress into this 'new world' together as a society. Therefore laws and regulations can hopefully keep up and we don't end up in a dystopia nightmare scenario.
      - deleted 9y ago
        [deleted]
      - SebNag_ 9y ago
        > laws and regulations can hopefully keep up I strongly doubt that.
  - justicezyx 9y ago
    Not sure what you want to say. US can do it open, China and Russia can still do it in secret. While, as a matter of fact, you will be naive if you believe that there is no secret research happening inside US government agencies.
    - hacker_9 9y ago
      My point is there is no stopping this research now, open or not neural nets just have too much potential. So I'm happy there is an open initiative that the big players support.
      - justicezyx 9y ago
- Danihan 9y ago
  You're not the only one who finds it scary, as there are massively popular books on the topic.. https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/1501227742 https://www.amazon.com/Superintelligence-Dangers-Strategies-...
  - fossuser 9y ago
    I found it difficult to make it through the first couple of chapters. Having just read The Gene, his analysis of the artificial selection option for super intelligence came across as very wrong - underestimating the complexity of polygenic traits and making (likely inaccurate) assumptions about their heredity. This type of thing has a dangerous history. The idea that a super intelligence could emerge from the internet unexplained also seemed pretty weak, but the first issue was so bad I found it hard to take anything else seriously (didn't trust the author's analysis). There are interesting issues with artificial consciousness, but I think they're in some ways similar to the issues with biological consciousness - the testing data the neural net is exposed to and it's underlying model can lead to minds that wouldn't be considered intelligent (and dangerous outcomes as a result).
    - briga 9y ago
      I would suggest giving it another go. There are few authors who have given this issue as much thought as Bostrom, and if the conclusions he draws are false at the very least that opens up the door for further conversation about the subject.
- tedsanders 9y ago
  No, I don't find this scary at all. My impression is that these algorithms are brittle, working only on very simple and structured problems. And these algorithms and their hyperparameters have to be specifically set by human researchers to match the structure of the problem. And if computer hardware progresses to reach orders of magnitude more performance, that may come at the cost of its present advantages in reliability, copying, and speed.
jboggan 9y ago
This is great. I've always thought intelligence can only be defined as an emergent property of self-replicating systems operating under stress, and this provides a good framework for that "stress".
- TeMPOraL 9y ago
  > intelligence can only be defined as an emergent property of self-replicating systems operating under stress That's a prescription for making one, but it isn't a good definition of intelligence. It doesn't tell us what to expect from it. Personally, I prefer intelligence defined as extremely strong, cross-domain optimization power.
mgummelt 9y ago
> a multiagent environment has no stable equilibrium It does. https://en.wikipedia.org/wiki/Evolutionarily_stable_strategy https://en.wikipedia.org/wiki/Evolutionarily_stable_strategy
- ilya_su 9y ago
  True in the general case, but the idea is to set things up (possibly by constantly changing the environment) so that there isn't one. This has happened at least once, when the brains of our ancestors have tripled in size relatievly quickly.
tomcam 9y ago
I was honestly hoping this would be about communications within the openai organization itself.
deepnet 9y ago
Peter Abeel ( OP link author with Igor Mordatch ) explains his groups work [1] as a guest lecturer for Berkeley's cs294-112 Deep Reinforcement Robotics. [1] https://www.youtube.com/watch?v=f4gKhK8Q6mY&list=PLkFD6_40KJIwTmSbCv9OVJB3YaO4sFwkX&index=25 https://www.youtube.com/watch?v=f4gKhK8Q6mY&list=PLkFD6_40KJ... This great talk starts with his work at OpenAI on neural net safety and adverserial images, then the OP research paper Emergence of Grounded Compositional Language in Multi-Agent Populations[2] concluding with his work ( with Andrew Ng ) reinforcement learning helicopter flight and stunt controllers from human pilots. The OP multi-agents divide labour and apparent collaborative plans appear. That the goal seeking agents split up, appear to dance to and from from their goal, distracting the predators from their kin is to all appearance coordinated and clever. Dawkin's Selfish Gene espouses alturism as an inevitability of genetic relatedness, the individual sacrifices but the genes persist in siblings. In this work alturism emerges purely memetically. The Nash equilibrium of cooperation jumps the local minima of selfishness in this prisoners dillemma. Multi agent enviroments are difficult to learn with many false minima for the learners. This work hints that the loose coupling of language, rather than direct sharing of memories or genes, is noisy enough to find more global solutions that appear complex or 'plan-like'. Maybe these AI's should be considered as planners, yet in a bottom-up immediate-heuristic emergent way. [2] https://arxiv.org/abs/1703.04908 https://arxiv.org/abs/1703.04908

199 points by maccaw 9y ago

Am I the only one that finds this scary? As I have repeatedly said, so far the intelligence we have been producing is NOT the kind that applies abstract logic rules to figure out meaning. It is the kind that takes full advantage of computers' strengths: perfect copying and speed. So good ideas are copied and propagate. Neural networks are just this on steroids... basically extracting signals from noise and doing a search in a space to maximize something, and storing the results. Humans were able to transmit knowledge, then produce books etc. Now bits can be perfectly copied with checksums. This isn't general intelligence in the human sense. But that's what makes it scary. It can solve these problems with brute force. Resistance soon may really be futile. Not just in running, avoiding capture and death. But also in ANY human system we rely on, including voting, due process of law, trust, reputation, family, sex, humor, etc.

zitterbewegung 9y ago
As someone that is teaching themselves Machine learning solving problems with "brute force" doesn't actually happen and Neural Networks are a way around using brute force. It seems that you are caught in the hype around machine learning. Systems that can copy ANY human system are very far off. Also, there are human systems where AI's would not be found to be acceptable.
- EGreg 9y ago
  Just to be clear: I didn't say they can copy any human system. I am saying they can find new ways to beat humans or groups of humans at the games we have set up. Chess and Go are just examples. Trust and Reputation are easily attacked. Voting is easily manipulated (look at gerrymandering and the increasingly shrill "fake news" claims.) Look, they are already better at making diagnoses than doctors. I am saying that the machines won't really "understand" the human meaning behind things. They will just have "deep" insight in having explored a huge amount of variations and extracted the combinations that maximize some goal. You think you are persuasive now? You think comedians are funny? Fast forrward 17 years when any software can deliver blistering putdowns to rappers, destroy the best comedian to the point where every human agrees the guy looks dull in comparison. The AI that can easily outcompete guys at wooing a girl online. And so on. Forget the Turing test. We are in for a future where programs simply "solve" the systems we set up faster and better than we can. And share that knowledge in compact form (such as artificial neural net calibration). Most of our systems rely on the inefficiency of an attacker. That they can't instantly start up 3,000,000 businesses or do sybil attacks in voting or destroy someone's credit overnight or win an election. But that is going to change. And I am saying that's just in the near future. Humor and trust is a little further.
  - cryoshon 9y ago
    yep. put simply: they will be better, once they are correctly configured. the task doesn't matter. there's nothing that's off limits, in the long run. the trick will be getting them to serve us on command, and to get them to hold back when we'd rather do something imperfectly and interpersonally rather than perfectly and computationally.
    - marcosdumay 9y ago
      We'll never be able to have them serve us, if for no other reason, because "we" have many diverse interests that are often antagonist between themselves. The best we can achieve is getting some of them to serve us on command, to protect us from other ones serving other people.
  - deleted 9y ago
    [deleted]
  - readonlydanger 9y ago
    Unfortunately, many of your assertions are inaccurate. As far as game playing goes, the ways in which AI agents beat human opponents is simply a matter of computers being able to process more information in a given period of time. The selection and evaluation criteria are actually pretty similar to human decision-making, if a bit more systematic. I can expand on this point in another comment, if you'd like. Chess and Go are not solved games, and will likely not be for some time (10+ years, for Go at least). Solving non-deterministic problems are even harder, and many likely will not be accomplished within our lifetimes[1]. There is no AI system that can generally make more accurate diagnoses than a physician. We're not there quite yet, and it will be a while. The best we've come are some pretty advanced expert systems, but these require very heavy input from physicians[2]. Please provide sources to validate your claims on the progress of AI. Extrapolation from data is dangerous, extrapolation from falsity is ignorance. As far as your take on understanding goes, the human "meaning" of things is inherently subjective. If we define "meaning" as some thing's value or role based on environmental context, then just like humans, any artificially intelligent system will only be able to determine meaning based on observations of the environment and both individual and collective experience. It's interesting that you seem to be focusing primarily on social domains that have inherently "human" contexts. Beyond misunderstanding the point of this paper, and the way AI systems work, I think you're missing the point. AI is, and for the time being will remain, an extension of the human mind. The decision-making needs to be developed (at some core level) by a human. Those goals need to be set by a human. The experiences and observational capabilities ultimately need to be determined by a human. Even AI systems that build other AI systems need to be directed to do so, and with strict goals, set by a human [3]. I highly suggest you take a moment to read openai's mission statement: https://openai.com/about/ https://openai.com/about/. AI is a tool. And like any powerful tool it must be used responsibly, freely, and openly. Openai is pursuing this goal and making efforts to ensure that this tool is available to as many people as possible to avoid the abuses implicit to your concerns. You obviously have an interest in AI and some knowledge of the field, but I worry your comments veer a bit towards fear-mongering. I suggest you use openai and resources like it to enrich your knowledge of both the advances and concerns of AI, because those are important, and we definitely need people thinking about these things. Ultimately, you are absolutely correct that eventually these systems will probably have the technical capability to influence elections, the economy, and more. But the only way they will is under the direction of humans. It it not the machine you should fear, but the man behind it. The same thing you ought to have been fearing all along. 1. http://fragrieu.free.fr/SearchingForSolutions.pdf http://fragrieu.free.fr/SearchingForSolutions.pdf 2. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1307157/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1307157/ 3. https://arxiv.org/abs/1611.02779 https://arxiv.org/abs/1611.02779
hacker_9 9y ago
Still, better this is done in the open. Otherwise Russia and China will just continue in secret, which is definitely a world I don't want to live in.
- EGreg 9y ago
  I agree it should be done in the open. Doesn't make it much less scary.
  - hacker_9 9y ago
    It does make it less scary because we progress into this 'new world' together as a society. Therefore laws and regulations can hopefully keep up and we don't end up in a dystopia nightmare scenario.
    - deleted 9y ago
      [deleted]
    - SebNag_ 9y ago
      > laws and regulations can hopefully keep up I strongly doubt that.
- justicezyx 9y ago
  Not sure what you want to say. US can do it open, China and Russia can still do it in secret. While, as a matter of fact, you will be naive if you believe that there is no secret research happening inside US government agencies.
  - hacker_9 9y ago
    My point is there is no stopping this research now, open or not neural nets just have too much potential. So I'm happy there is an open initiative that the big players support.
    - justicezyx 9y ago
Danihan 9y ago
You're not the only one who finds it scary, as there are massively popular books on the topic.. https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/1501227742 https://www.amazon.com/Superintelligence-Dangers-Strategies-...
- fossuser 9y ago
  I found it difficult to make it through the first couple of chapters. Having just read The Gene, his analysis of the artificial selection option for super intelligence came across as very wrong - underestimating the complexity of polygenic traits and making (likely inaccurate) assumptions about their heredity. This type of thing has a dangerous history. The idea that a super intelligence could emerge from the internet unexplained also seemed pretty weak, but the first issue was so bad I found it hard to take anything else seriously (didn't trust the author's analysis). There are interesting issues with artificial consciousness, but I think they're in some ways similar to the issues with biological consciousness - the testing data the neural net is exposed to and it's underlying model can lead to minds that wouldn't be considered intelligent (and dangerous outcomes as a result).
  - briga 9y ago
    I would suggest giving it another go. There are few authors who have given this issue as much thought as Bostrom, and if the conclusions he draws are false at the very least that opens up the door for further conversation about the subject.
tedsanders 9y ago
No, I don't find this scary at all. My impression is that these algorithms are brittle, working only on very simple and structured problems. And these algorithms and their hyperparameters have to be specifically set by human researchers to match the structure of the problem. And if computer hardware progresses to reach orders of magnitude more performance, that may come at the cost of its present advantages in reliability, copying, and speed.

jboggan 9y ago

This is great. I've always thought intelligence can only be defined as an emergent property of self-replicating systems operating under stress, and this provides a good framework for that "stress".

TeMPOraL 9y ago
> intelligence can only be defined as an emergent property of self-replicating systems operating under stress That's a prescription for making one, but it isn't a good definition of intelligence. It doesn't tell us what to expect from it. Personally, I prefer intelligence defined as extremely strong, cross-domain optimization power.

mgummelt 9y ago

> a multiagent environment has no stable equilibrium It does. https://en.wikipedia.org/wiki/Evolutionarily_stable_strategy https://en.wikipedia.org/wiki/Evolutionarily_stable_strategy

ilya_su 9y ago
True in the general case, but the idea is to set things up (possibly by constantly changing the environment) so that there isn't one. This has happened at least once, when the brains of our ancestors have tripled in size relatievly quickly.

tomcam 9y ago

I was honestly hoping this would be about communications within the openai organization itself.

deepnet 9y ago

Peter Abeel ( OP link author with Igor Mordatch ) explains his groups work [1] as a guest lecturer for Berkeley's cs294-112 Deep Reinforcement Robotics. [1] https://www.youtube.com/watch?v=f4gKhK8Q6mY&list=PLkFD6_40KJIwTmSbCv9OVJB3YaO4sFwkX&index=25 https://www.youtube.com/watch?v=f4gKhK8Q6mY&list=PLkFD6_40KJ... This great talk starts with his work at OpenAI on neural net safety and adverserial images, then the OP research paper Emergence of Grounded Compositional Language in Multi-Agent Populations[2] concluding with his work ( with Andrew Ng ) reinforcement learning helicopter flight and stunt controllers from human pilots. The OP multi-agents divide labour and apparent collaborative plans appear. That the goal seeking agents split up, appear to dance to and from from their goal, distracting the predators from their kin is to all appearance coordinated and clever. Dawkin's Selfish Gene espouses alturism as an inevitability of genetic relatedness, the individual sacrifices but the genes persist in siblings. In this work alturism emerges purely memetically. The Nash equilibrium of cooperation jumps the local minima of selfishness in this prisoners dillemma. Multi agent enviroments are difficult to learn with many false minima for the learners. This work hints that the loose coupling of language, rather than direct sharing of memories or genes, is noisy enough to find more global solutions that appear complex or 'plan-like'. Maybe these AI's should be considered as planners, yet in a bottom-up immediate-heuristic emergent way. [2] https://arxiv.org/abs/1703.04908 https://arxiv.org/abs/1703.04908