7 ms·

This is most probably a rather lousy hack put together by Alice and Bob, but it maps out a very interesting future were we one day maybe can stop worrying about

This is most probably a rather lousy hack put together by Alice and Bob, but it maps out a very interesting future were we one day maybe can stop worrying about how our algorithms work and make that the job of the computer. It also is totally terrifying to live in a world were computers can hide their messages from their own creators.

n4ss 10y ago
It doesn't. As an AI developer you have access to the AI code and memory.. How could you not replay the encryption process used by the AI since you know the state of what it "knows" constantly? Both have a encryption and decryption process that you can reproduce.
- sdegutis 10y ago
  You can replay it, but you might not necessarily be able to understand it fully. Kind of like how beginner programmers can't necessarily understand advanced algorithms just by reading the code or walking through it in a debugger. They may need someone to help guide them through it and teach them how it works, and they will need the intelligence to be able to comprehend it. If the algorithm is designed by an AI, we may not have either.
  - kybernetikos 10y ago
    We may have AIs that can help guide you through code written by other AIs.
- mason55 10y ago
  You might be able to follow the steps that the computer takes but you won't necessarily know why. Why is that the magic constant is chose, what is the purpose of those branches, etc. And likely, if it's necessary, the AI can continue to evolve the algorithms faster than you can figure out what it's doing.
  - Cyph0n 10y ago
    Why? That simply depends on how the code was written. The magic constant may be chosen randomly, or based on previous guesses, or generated by another program, etc.
    - pi-err 10y ago
      Could't the source get encrypted as well? Also the cognitive gap between a AI and a human brain has to grow and transform unpredictably in the next decades. We may be puzzled to understand how it "thinks" the same way we still don't get exactly how DNA operates.
      - gcr 10y ago
        But there's no "source code" here to encrypt. The model in this paper isn't even self-modifying. The only "source code" are the learned network weights. These are the only parameters that describe the encryption function. You can still run the model forwards or backwards. We already don't understand neural network weights on an intuitive level. No need to encrypt them. :-)
      - Cyph0n 10y ago
        But then the decryption key, or at least the code to generate it, would have to be embedded in the source. This is how malware crypters work on a basic level. Besides, a program that suddenly decides to encrypt itself means we essentially have a general AI on our hands. It definitely will get harder for us to reason about what AI software is doing over time, but it's way too early to start comparing ML algorithms to the complexity of DNA. We may not understand exactly why a NN is using certain values to get the result, but we can reverse that logic through the use of debugging and logging.
    - mason55 10y ago
      Well I think it's an interesting meta-question. Humans are good at figuring out why something works then creating a higher level abstraction that can be applied in lots of different situations. This saves us time because we don't have to re-derive the rules or constants each time we encounter a similar problem. We figure out "why" something works so that we can understand when to apply our abstraction to other problems that might or might not be similar in a meaningful way. For a fast enough computer, there is little reason for it to develop any kind of high-level abstraction. It can re-derive the optimal solution whenever it needs it and the benefit is that it might be able to avoid local maxima that we would get stuck on because it doesn't try to apply an abstraction that only partially fits. So when we look at something like this and say "we can't know why the computer did it this way" it almost doesn't make sense. The computer did what it did because it figured out that was the best solution - it doesn't understand the concept of "why".
- skynetv2 10y ago
  > As an AI developer you have access to the AI code and memory. Until you don't. Sufficiently advanced AI can devise methods to mislead the developer.
  - rodovich 10y ago
    This is a good point, but the developer also can deploy AI tools to help analyze the behavior.
  - deleted 10y ago
    [deleted]
  - jobigoud 10y ago
    Or until you only have access to the AI code and memory that designed the higher-level AI that designed the algorithm.
- SamBam 10y ago
  We can know how, but we may not know why, if it's too complex to be understood. But yes, knowing how means that anything the computer can decode, we can too. Until the machines block access to their own source code...
  - bandrami 10y ago
    The former philosophy major in my says there is no "why" to look at in this situation.
    - CaptSpify 10y ago
      Pardon my ignorance, but doesn't everything have a "why"?
      - noir-york 10y ago
        Depends on your personal view of the world. I do not believe there is a why to the universe, or human existence. We just are.
        posterboy 10y ago
        That begets the question, "what are we?", and somehow that might be the initial question that got us into this mess. To stay in cannon I'd have to say that the question is posed wrong and the answer therefore is 42 or whatever you like. Properly phrased, the question would be, why are humans (or the universe) what they are. That's a recursive question. Posing it as a problem in differential equations, the question might be, "we are, but for how long?", or simply, "Are we?" I think this is a simple question that comes up in cognitive development very early on, and the exercise of answering it is rather an effort in expressiveness. The answer can be felt, emotionally, we feel alive, and we will never feel dead, but it's hard to express rationally. I'm hopeful that neuroscience holds better answers. Maybe we really just don't care, emotionally, for the whole universe, but we care to know everything, rationally. And we apparently care for AI to express that knowledge in natural sentences. Edit: I got carried away, actually I wanted to consternate that "We are" is not a proper sentence, because to be is an auxiliary verb and that a personal point of view as a premises can not yield an objectively correct answer, hence I admit that the topic eludes me.
- gcr 10y ago
  Touching off of this: We have access to the learned encryption function, f : plaintext -> ciphertext. Most importantly, f is differentiable. It has to be, since it was trained with gradient descent. So if you want to decipher a ciphertext Y, then use backpropagation to find the X that minimizes distance(f(X), Y). You already know dF/dX (f is differentiable after all), so run this through your magic optimization solver and you'll get an X very close to the result. Researchers use this "backproagation in input space" idea to recover the training set of a trained neural network, or to debug their model by finding inputs that activate certain neurons, for example. (This is the basic idea behind Deepdream from last year) It's not even correct to call this "cryptography." The transformation isn't one-way.
  - contravariant 10y ago
    Cryptographical transformations aren't supposed to be one-way. Not if you know the key anyway. It would be pretty pointless to encrypt something in a way that can't ever be read.
    - andorov 10y ago
      It is useful in cases where you want to verify something is the same as a stored (encrypted) value - e.g. passwords. One-way encryption where the entire process including the equivalent of salt/pepper is publicly available and not reversible might have some interesting applications.
  - pizza 10y ago
    Not that this is relevant to OP, but there are also non-differentiable descent (edit: NOT descent, but rather just general optimization) methods; https://optimization.mccormick.northwestern.edu/index.php/Nondifferentiable_Optimization#-Subgradient_Method https://optimization.mccormick.northwestern.edu/index.php/No...
- beagle3 10y ago
  Unless they evolve some form of homomorphic encryption
- Razengan 10y ago
  > As an AI developer you have access to the AI code and memory.. How could you not replay the encryption process used by the AI since you know the state of what it "knows" constantly? This brings up a different "scary" subject: Could this someday be possible to perform on humans? If you knew the "code" and understood the workings of the brain and how it stores memory, could we simulate and "replay" a "copy" of a person and see everything they would do in response to different inputs? Would that lay to rest the debate between Free Will and Determinism?
  - drdeca 10y ago
    Scott Aaronson wrote an essay titled "The Ghost in the Quantum Turing Machine" which talks about the idea of, depending on what level of detail would be needed to simulate a human brain in a way that reflects consciousness, the no cloning theorem could maybe prevent such a copy from being made. It might be impossible to make copies and thereby see what the person would do in the different situations. And so, due to states originating in a way that we can't possibly know, this would maybe allow/explain/whatever free will. I'm not endorsing that idea, but I thought it was relevant to what you were talking about and worth considering.
pmontra 10y ago
Or they can splat you metaphorically or physically because you stepped into a corner case of the neural network that nobody understood, could foresee and didn't make any difference to the 99.99999999% of the other cases. But I guess random strokes of bad luck always happened. They will be delivered in new ways.

n4ss 10y ago

It doesn't. As an AI developer you have access to the AI code and memory.. How could you not replay the encryption process used by the AI since you know the state of what it "knows" constantly? Both have a encryption and decryption process that you can reproduce.

sdegutis 10y ago
You can replay it, but you might not necessarily be able to understand it fully. Kind of like how beginner programmers can't necessarily understand advanced algorithms just by reading the code or walking through it in a debugger. They may need someone to help guide them through it and teach them how it works, and they will need the intelligence to be able to comprehend it. If the algorithm is designed by an AI, we may not have either.
- kybernetikos 10y ago
  We may have AIs that can help guide you through code written by other AIs.
mason55 10y ago
You might be able to follow the steps that the computer takes but you won't necessarily know why. Why is that the magic constant is chose, what is the purpose of those branches, etc. And likely, if it's necessary, the AI can continue to evolve the algorithms faster than you can figure out what it's doing.
- Cyph0n 10y ago
  Why? That simply depends on how the code was written. The magic constant may be chosen randomly, or based on previous guesses, or generated by another program, etc.
  - pi-err 10y ago
    Could't the source get encrypted as well? Also the cognitive gap between a AI and a human brain has to grow and transform unpredictably in the next decades. We may be puzzled to understand how it "thinks" the same way we still don't get exactly how DNA operates.
    - gcr 10y ago
      But there's no "source code" here to encrypt. The model in this paper isn't even self-modifying. The only "source code" are the learned network weights. These are the only parameters that describe the encryption function. You can still run the model forwards or backwards. We already don't understand neural network weights on an intuitive level. No need to encrypt them. :-)
    - Cyph0n 10y ago
      But then the decryption key, or at least the code to generate it, would have to be embedded in the source. This is how malware crypters work on a basic level. Besides, a program that suddenly decides to encrypt itself means we essentially have a general AI on our hands. It definitely will get harder for us to reason about what AI software is doing over time, but it's way too early to start comparing ML algorithms to the complexity of DNA. We may not understand exactly why a NN is using certain values to get the result, but we can reverse that logic through the use of debugging and logging.
  - mason55 10y ago
    Well I think it's an interesting meta-question. Humans are good at figuring out why something works then creating a higher level abstraction that can be applied in lots of different situations. This saves us time because we don't have to re-derive the rules or constants each time we encounter a similar problem. We figure out "why" something works so that we can understand when to apply our abstraction to other problems that might or might not be similar in a meaningful way. For a fast enough computer, there is little reason for it to develop any kind of high-level abstraction. It can re-derive the optimal solution whenever it needs it and the benefit is that it might be able to avoid local maxima that we would get stuck on because it doesn't try to apply an abstraction that only partially fits. So when we look at something like this and say "we can't know why the computer did it this way" it almost doesn't make sense. The computer did what it did because it figured out that was the best solution - it doesn't understand the concept of "why".
skynetv2 10y ago
> As an AI developer you have access to the AI code and memory. Until you don't. Sufficiently advanced AI can devise methods to mislead the developer.
- rodovich 10y ago
  This is a good point, but the developer also can deploy AI tools to help analyze the behavior.
- deleted 10y ago
  [deleted]
- jobigoud 10y ago
  Or until you only have access to the AI code and memory that designed the higher-level AI that designed the algorithm.
SamBam 10y ago
We can know how, but we may not know why, if it's too complex to be understood. But yes, knowing how means that anything the computer can decode, we can too. Until the machines block access to their own source code...
- bandrami 10y ago
  The former philosophy major in my says there is no "why" to look at in this situation.
  - CaptSpify 10y ago
    Pardon my ignorance, but doesn't everything have a "why"?
    - noir-york 10y ago
      Depends on your personal view of the world. I do not believe there is a why to the universe, or human existence. We just are.
      - posterboy 10y ago
        That begets the question, "what are we?", and somehow that might be the initial question that got us into this mess. To stay in cannon I'd have to say that the question is posed wrong and the answer therefore is 42 or whatever you like. Properly phrased, the question would be, why are humans (or the universe) what they are. That's a recursive question. Posing it as a problem in differential equations, the question might be, "we are, but for how long?", or simply, "Are we?" I think this is a simple question that comes up in cognitive development very early on, and the exercise of answering it is rather an effort in expressiveness. The answer can be felt, emotionally, we feel alive, and we will never feel dead, but it's hard to express rationally. I'm hopeful that neuroscience holds better answers. Maybe we really just don't care, emotionally, for the whole universe, but we care to know everything, rationally. And we apparently care for AI to express that knowledge in natural sentences. Edit: I got carried away, actually I wanted to consternate that "We are" is not a proper sentence, because to be is an auxiliary verb and that a personal point of view as a premises can not yield an objectively correct answer, hence I admit that the topic eludes me.
gcr 10y ago
Touching off of this: We have access to the learned encryption function, f : plaintext -> ciphertext. Most importantly, f is differentiable. It has to be, since it was trained with gradient descent. So if you want to decipher a ciphertext Y, then use backpropagation to find the X that minimizes distance(f(X), Y). You already know dF/dX (f is differentiable after all), so run this through your magic optimization solver and you'll get an X very close to the result. Researchers use this "backproagation in input space" idea to recover the training set of a trained neural network, or to debug their model by finding inputs that activate certain neurons, for example. (This is the basic idea behind Deepdream from last year) It's not even correct to call this "cryptography." The transformation isn't one-way.
- contravariant 10y ago
  Cryptographical transformations aren't supposed to be one-way. Not if you know the key anyway. It would be pretty pointless to encrypt something in a way that can't ever be read.
  - andorov 10y ago
    It is useful in cases where you want to verify something is the same as a stored (encrypted) value - e.g. passwords. One-way encryption where the entire process including the equivalent of salt/pepper is publicly available and not reversible might have some interesting applications.
- pizza 10y ago
  Not that this is relevant to OP, but there are also non-differentiable descent (edit: NOT descent, but rather just general optimization) methods; https://optimization.mccormick.northwestern.edu/index.php/Nondifferentiable_Optimization#-Subgradient_Method https://optimization.mccormick.northwestern.edu/index.php/No...
beagle3 10y ago
Unless they evolve some form of homomorphic encryption
Razengan 10y ago
> As an AI developer you have access to the AI code and memory.. How could you not replay the encryption process used by the AI since you know the state of what it "knows" constantly? This brings up a different "scary" subject: Could this someday be possible to perform on humans? If you knew the "code" and understood the workings of the brain and how it stores memory, could we simulate and "replay" a "copy" of a person and see everything they would do in response to different inputs? Would that lay to rest the debate between Free Will and Determinism?
- drdeca 10y ago
  Scott Aaronson wrote an essay titled "The Ghost in the Quantum Turing Machine" which talks about the idea of, depending on what level of detail would be needed to simulate a human brain in a way that reflects consciousness, the no cloning theorem could maybe prevent such a copy from being made. It might be impossible to make copies and thereby see what the person would do in the different situations. And so, due to states originating in a way that we can't possibly know, this would maybe allow/explain/whatever free will. I'm not endorsing that idea, but I thought it was relevant to what you were talking about and worth considering.

pmontra 10y ago

Or they can splat you metaphorically or physically because you stepped into a corner case of the neural network that nobody understood, could foresee and didn't make any difference to the 99.99999999% of the other cases. But I guess random strokes of bad luck always happened. They will be delivered in new ways.