4 ms·

The way I read the title was: \forall messages M s.t. \forall decontaminating functions F, F does not decontaminate M. I think what the paper actually shows is

The way I read the title was: \forall messages M s.t. \forall decontaminating functions F, F does not decontaminate M. I think what the paper actually shows is \exists message M, s.t. \forall decontaminating functions F, F does not decontaminate M. In fact, we know the title (as I interpreted it) is false - the paper gives an example of a message that can be read as plain text which can be truly decontaminated. Basically, when we get a message, we'll be able to immediately tell whether we can decontaminate it or not. If not, the paper argues we're doomed because eventually it'll leak. I just don't find that to be very interesting, and definitely not as strong a claim as the title (as I interpreted it).

pdonis 9y ago
> when we get a message, we'll be able to immediately tell whether we can decontaminate it or not I don't think this is true (at least not "immediately"). The paper is arguing that we will need computers to help decode any non-trivial message, and any computer program capable of decoding a non-trivial message will be Turing complete, and hence capable of running virus code that could have damaging or destructive effects. (The paper's example was a message containing equations in LaTeX, which is Turing complete, hence an ETI could embed a virus in LaTeX code, and humans might not be able to spot it.) The strategy the paper is advocating, as far as I can see, is that, unless the message is simple enough that humans can decode it without computer help, and thereby check directly whether it contains any dangerous information, the message should be destroyed. I suppose this could count as "we can't tell whether we can decontaminate it", but I'm not sure that's what you meant.
- IntronExon 9y ago
  I suppose one option is to stick a computer, not networked, on a small spacecraft along with a single astronaut and pop them into LEO. Run the code, and have the astronaut tell us what it says or does. If something goes wrong, the whole thing deorbits. If we’re really paranoid we could send it well past LEO. Edit: If we were incredibly paranoid and had some self-sacrificing astronauts it could be a one-way trip to Mercury.
  - nordsieck 9y ago
    It sounds like you've never heard of the AI Box Experiment[0]. I personally don't believe that there is any way to safely do isolation while still maintaining the potential to transmit the message. [0] http://yudkowsky.net/singularity/aibox/ http://yudkowsky.net/singularity/aibox/
    - IntronExon 9y ago
      We could restrict the astronaut to using Morse code, and have literally no other data link back to Earth. The astronaut can be immobilized except for one finger used to key the Morse code. The ship can be little more than a ballistic missile set to run a set number of orbits before burning in the atmosphere. There is no way out of that box, even if we want to let it out.
      - adrianN 9y ago
        Things like artificial intelligences can be transferred via Morse code.
        IntronExon 9y ago
        Yes, but not through whatever a human can tap out in a relatively short period of time. Besides, you don’t need to receive the signal a computer, just a speaker and a guy with a pen and paper.
      - sbierwagen 9y ago
        There's no way for you to get the astronaut out of the box. But consider a hostile, superhuman AI. Who knows what it can do? But more to the point, who knows what it can convince the astronaut to do? Can it convince the astronaut to transmit a hostile message back to Earth, suborning the receiving computer, and thereby letting the AI out of the box? In the AI alignment field, this is called the "value drift" problem.
        IntronExon 9y ago

pdonis 9y ago

> when we get a message, we'll be able to immediately tell whether we can decontaminate it or not I don't think this is true (at least not "immediately"). The paper is arguing that we will need computers to help decode any non-trivial message, and any computer program capable of decoding a non-trivial message will be Turing complete, and hence capable of running virus code that could have damaging or destructive effects. (The paper's example was a message containing equations in LaTeX, which is Turing complete, hence an ETI could embed a virus in LaTeX code, and humans might not be able to spot it.) The strategy the paper is advocating, as far as I can see, is that, unless the message is simple enough that humans can decode it without computer help, and thereby check directly whether it contains any dangerous information, the message should be destroyed. I suppose this could count as "we can't tell whether we can decontaminate it", but I'm not sure that's what you meant.

IntronExon 9y ago
I suppose one option is to stick a computer, not networked, on a small spacecraft along with a single astronaut and pop them into LEO. Run the code, and have the astronaut tell us what it says or does. If something goes wrong, the whole thing deorbits. If we’re really paranoid we could send it well past LEO. Edit: If we were incredibly paranoid and had some self-sacrificing astronauts it could be a one-way trip to Mercury.
- nordsieck 9y ago
  It sounds like you've never heard of the AI Box Experiment[0]. I personally don't believe that there is any way to safely do isolation while still maintaining the potential to transmit the message. [0] http://yudkowsky.net/singularity/aibox/ http://yudkowsky.net/singularity/aibox/
  - IntronExon 9y ago
    We could restrict the astronaut to using Morse code, and have literally no other data link back to Earth. The astronaut can be immobilized except for one finger used to key the Morse code. The ship can be little more than a ballistic missile set to run a set number of orbits before burning in the atmosphere. There is no way out of that box, even if we want to let it out.
    - adrianN 9y ago
      Things like artificial intelligences can be transferred via Morse code.
      - IntronExon 9y ago
        Yes, but not through whatever a human can tap out in a relatively short period of time. Besides, you don’t need to receive the signal a computer, just a speaker and a guy with a pen and paper.
    - sbierwagen 9y ago
      There's no way for you to get the astronaut out of the box. But consider a hostile, superhuman AI. Who knows what it can do? But more to the point, who knows what it can convince the astronaut to do? Can it convince the astronaut to transmit a hostile message back to Earth, suborning the receiving computer, and thereby letting the AI out of the box? In the AI alignment field, this is called the "value drift" problem.
      - IntronExon 9y ago