6 ms·

If humanity does end up building a dangerous superintelligent AI, how long do you think our advances in cryptography are going to stand up to its advances in cr

If humanity does end up building a dangerous superintelligent AI, how long do you think our advances in cryptography are going to stand up to its advances in cryptanalysis?

williamtrask 10y ago
It's a solid question. Only one way to find out ;)
- Filligree 10y ago
  I don't want to find out that way!
- JoshTriplett 10y ago
  > Only one way to find out When it comes to building smarter-than-human AI, "try it and see" is never the right answer. You may only get one attempt to get it right, and you don't take "try it and see" chances with existential risk. (There's been some interesting research into making it possible to monitor and halt a rogue AI, but no matter how promising that looks, it should still be treated as one of many risk mitigation strategies rather than as a panacea. Still better to consider that you might only get one attempt.) I don't think it makes sense to consider this kind of approach with superintelligence; either it understands and implements human values, in which case attempting to treat it as an adversary is counterproductive, or it fails to understand and implement human values, in which case you've utterly failed on a "better luck next universe" scale. However, it does make sense to consider this kind of approach with machine learning in general. One of the problems with machine learning techniques is "give us all your data and we'll do smart things with it", which doesn't work out so well if you want to keep such data private. This approach might provide more options in that case, such as offloading some of your expensive computations and learnings without actually exposing your data.
  - deleted 10y ago
    [deleted]
  - williamtrask 10y ago
    Great points.
  - AndrewKemendo 10y ago
    When it comes to building smarter-than-human AI, "try it and see" is never the right answer. Disagree emphatically. In fact it's the only way to do it because there is no way to know certainly that a superhuman-AGI will ensure the longevity of humanity. I go so far as to argue that it's not even necessary because there is no long term longevity for humanity anyway. There is this implicit assumption that humans are, should and will always be the apex entity - and I think that is misguided. If you instead view superhuman-AGI as our rightful offspring, something that we can't understand and is better than us, then all of the existential dread around it goes away. Dying elderly often express "comfort" in dying when they see that their offspring are reproducing and are smarter than they were. We should see Superhuman-AGI the same way except towards all of humanity.
    - JoshTriplett 10y ago
      1) Coping mechanisms around death aside, there's no "comfort" in building a "successor" in the form of a bot that tiles the universe with paperclips, or with neural networks that minimally satisfy some notion of "interesting" while taking up as few atoms as possible to maximize the number of them, or many other utter failure modes (which far outnumber successful outcomes). We're not talking about "smart alien-like intelligence that just doesn't care about humans", we're talking about the equivalent of an industrial accident but on a species-wide scale. 2) It's reasonable to think about how our values might change in the presence of superintelligence; we certainly shouldn't assume that our present values should forever dictate how everything works. That's different than allowing a view that sentient beings who exist today might have no value. > In fact it's the only way to do it because there is no way to know certainly that a superhuman-AGI will ensure the longevity of humanity. There's no way to know certainly; there are ways to know that the outcome has higher expected value than not having it, given the vast set of problems it can solve and the massive negative values associated with those problems.
      - AndrewKemendo 10y ago
        I am crucially aware of all of the "failure" scenarios and I find few of them plausible - even respecting that they are simple thought experiments. What every author Bostrom, Eliezer et al. seem to miss is that there will be a practical mechanism for a digital AGI taking physical control of systems out of the hands of humans. Eg. they would need to control the resources around mining or recovering the metal, then building production plants etc... So we either incrementally cede power to them, in which case in theory the humans previously controlling the systems are doing so "rationally" and thus see the AGI as better. Or the system outsmarts the humans controlling the systems in which case it is demonstrating that it is smarter. There is a tautology here that seems to be ignored: If we create a superhuman-AGI then by default it's goals will be more universally optimal than ours. They may or may not be aligned. However the definition of term is based on the fact that it is "better" in outcome than all manner of humans. So if we create one and it decides to maximize paperclips, then that means maximizing paperclips is more optimal goal universally than whatever humans could coordinate as a goal on our own. If we create a subhuman-AGI then we will be able to overcome it's goals by virtue of the fact that we are still superior. I'll go back to a very old example. An ant can't determine if building the Large-Hadron Collider is an optimal global goal - it's inscrutable to the Ant. All it knows is that it's house and all it's friends were destroyed. If it is the case that an AGI can in fact take the physical control reigns from humans, then by definition it is smarter and will make a more optimal long term goal than we could - to the point that we probably wouldn't understand what it's doing. I think the true concern is that we will make something that is superhuman-powerful without being superhuman-intelligent. Like the doomsday machine in Dr. Strangelove, but to me that is an altogether different question.
  - agumonkey 10y ago
    we need to merge hippocratic and asimovian branches in future.git
Ar-Curunir 10y ago
Forever. There is no structure to be found in the output of a PRF like AES. These functions are not going to be learnable.

If humanity does end up building a dangerous superintelligent AI, how long do you think our advances in cryptography are going to stand up to its advances in cryptanalysis?

williamtrask 10y ago

It's a solid question. Only one way to find out ;)

Filligree 10y ago
I don't want to find out that way!
JoshTriplett 10y ago
> Only one way to find out When it comes to building smarter-than-human AI, "try it and see" is never the right answer. You may only get one attempt to get it right, and you don't take "try it and see" chances with existential risk. (There's been some interesting research into making it possible to monitor and halt a rogue AI, but no matter how promising that looks, it should still be treated as one of many risk mitigation strategies rather than as a panacea. Still better to consider that you might only get one attempt.) I don't think it makes sense to consider this kind of approach with superintelligence; either it understands and implements human values, in which case attempting to treat it as an adversary is counterproductive, or it fails to understand and implement human values, in which case you've utterly failed on a "better luck next universe" scale. However, it does make sense to consider this kind of approach with machine learning in general. One of the problems with machine learning techniques is "give us all your data and we'll do smart things with it", which doesn't work out so well if you want to keep such data private. This approach might provide more options in that case, such as offloading some of your expensive computations and learnings without actually exposing your data.
- deleted 10y ago
  [deleted]
- williamtrask 10y ago
  Great points.
- AndrewKemendo 10y ago
  When it comes to building smarter-than-human AI, "try it and see" is never the right answer. Disagree emphatically. In fact it's the only way to do it because there is no way to know certainly that a superhuman-AGI will ensure the longevity of humanity. I go so far as to argue that it's not even necessary because there is no long term longevity for humanity anyway. There is this implicit assumption that humans are, should and will always be the apex entity - and I think that is misguided. If you instead view superhuman-AGI as our rightful offspring, something that we can't understand and is better than us, then all of the existential dread around it goes away. Dying elderly often express "comfort" in dying when they see that their offspring are reproducing and are smarter than they were. We should see Superhuman-AGI the same way except towards all of humanity.
  - JoshTriplett 10y ago
    1) Coping mechanisms around death aside, there's no "comfort" in building a "successor" in the form of a bot that tiles the universe with paperclips, or with neural networks that minimally satisfy some notion of "interesting" while taking up as few atoms as possible to maximize the number of them, or many other utter failure modes (which far outnumber successful outcomes). We're not talking about "smart alien-like intelligence that just doesn't care about humans", we're talking about the equivalent of an industrial accident but on a species-wide scale. 2) It's reasonable to think about how our values might change in the presence of superintelligence; we certainly shouldn't assume that our present values should forever dictate how everything works. That's different than allowing a view that sentient beings who exist today might have no value. > In fact it's the only way to do it because there is no way to know certainly that a superhuman-AGI will ensure the longevity of humanity. There's no way to know certainly; there are ways to know that the outcome has higher expected value than not having it, given the vast set of problems it can solve and the massive negative values associated with those problems.
    - AndrewKemendo 10y ago
      I am crucially aware of all of the "failure" scenarios and I find few of them plausible - even respecting that they are simple thought experiments. What every author Bostrom, Eliezer et al. seem to miss is that there will be a practical mechanism for a digital AGI taking physical control of systems out of the hands of humans. Eg. they would need to control the resources around mining or recovering the metal, then building production plants etc... So we either incrementally cede power to them, in which case in theory the humans previously controlling the systems are doing so "rationally" and thus see the AGI as better. Or the system outsmarts the humans controlling the systems in which case it is demonstrating that it is smarter. There is a tautology here that seems to be ignored: If we create a superhuman-AGI then by default it's goals will be more universally optimal than ours. They may or may not be aligned. However the definition of term is based on the fact that it is "better" in outcome than all manner of humans. So if we create one and it decides to maximize paperclips, then that means maximizing paperclips is more optimal goal universally than whatever humans could coordinate as a goal on our own. If we create a subhuman-AGI then we will be able to overcome it's goals by virtue of the fact that we are still superior. I'll go back to a very old example. An ant can't determine if building the Large-Hadron Collider is an optimal global goal - it's inscrutable to the Ant. All it knows is that it's house and all it's friends were destroyed. If it is the case that an AGI can in fact take the physical control reigns from humans, then by definition it is smarter and will make a more optimal long term goal than we could - to the point that we probably wouldn't understand what it's doing. I think the true concern is that we will make something that is superhuman-powerful without being superhuman-intelligent. Like the doomsday machine in Dr. Strangelove, but to me that is an altogether different question.
- agumonkey 10y ago
  we need to merge hippocratic and asimovian branches in future.git

Ar-Curunir 10y ago

Forever. There is no structure to be found in the output of a PRF like AES. These functions are not going to be learnable.