5 ms·
> The entire paper assumes we get a certain type of communication and in the course of decoding it destroy humanity. "Assume" is a bit of an odd way of putting
by shkkmo 9y ago
> The entire paper assumes we get a certain type of communication and in the course of decoding it destroy humanity.
"Assume" is a bit of an odd way of putting it. The paper is a discussion of the risks of receiving a certain type of message.
> but the paper presents no proof whatsoever that "message decontamination is impossible."
That's a bit harsh. The paper provides several arguments which I find quite solid. Which of them did you find unconvincing? The existence of message that can't be parsed by hand? The risk of manipulative AI running on an air-gapped computer? The difficulty of containing a message given that we can't turn off the source? The difficulty of containing knowledge/information over longer time scales?
I didn't think there was much particularly new in that article. It's mostly a re-hashing of other people's arguments. The most interesting point the article made is this conclusion:
"As we realize that some message types are potentially
dangerous, we can adapt our own peaceful transmissions
accordingly. We should certainly not transmit any code.
Instead, a plain text encyclopedia (Heidmann 1993), images,
music etc. in a simple format are adequate. No
advanced computer should be required to decrypt our
message."
- wyattpeak 9y agoPersonally, I find the idea that "humans could be bargained into a poor position" to be fairly weak. It's certainly possible, but negotiations with hidden agendas have been part and parcel of humanity for all time, and are a major basis for all foreign (and most internal) affairs. The paper presents no predicament that China couldn't put the US in tomorrow with a suitably (apparently) one-sided deal, and yet countries don't as a rule isolate themselves to avoid hearing potentially deadly trade offers.
- pdonis 9y ago> I find the idea that "humans could be bargained into a poor position" to be fairly weak That's because you're assuming that whatever ETI or AI we are bargaining with has roughly our intelligence. That is what has made it possible for humans to negotiate with each other instead of isolating themselves to avoid hearing potentially deadly offers. But any ETI or AI capable of sending us a complex message will probably be much more intelligent than we are. In that case, we would basically be in the same position as, say, a dog trying to negotiate with humans; we simply would not even be able to comprehend what the other side was doing or thinking.
- wyattpeak 9y agoNo, I'm not, I'm saying that magic intelligence doesn't change the rules of the game. If I were negotiating with an alien intelligence, I would assume that it could think circles around me, so I wouldn't be about to accept any bargain that I couldn't - fully - think through the ramifications of. I'm not going to accept complicated bargains, only extremely simple ones. This would most probably result in making no bargains at all. So be it. I reject the notion that a being capable of reason, holding all the cards, is unable to create an unwinnable state for its opponent, when inaction is a possibility. The most intelligent computer in the universe can't beat you at chess if you don't play.
- TheOtherHobbes 9y agoHow would you know you’re playing chess, or even that you’re negotiating? Any sufficiently advanced alien invasion could be indistinguishable from local politics. I think the paper fails because it assumes a magical hypersmart AI would arrive as a message at a radio telescope. But if you’re going to magically assume a magical hypersmart AI, you may as well assume it can get here magically without being noticed.
- jacobush 9y agoI wonder from an anthropological view, how these discussions relate to those of the past, about demons.
- wyattpeak 9y agoAgreed, there are a million more convincing ways an alien intelligence could take over, most likely - as you say - without us ever noticing. I'm just jumping off from the particular assumption that the paper makes.
- pdonis 9y ago> I reject the notion that a being capable of reason, holding all the cards, is unable to create an unwinnable state for its opponent, when inaction is a possibility. If this is true for you, it's also true for the AI. Which would be a contradiction, similar to the old classic irresistible force meeting immovable object. So the notion you're rejecting is the only consistent possibility.
- obblekk 9y agoThe way I read the title was: \forall messages M s.t. \forall decontaminating functions F, F does not decontaminate M. I think what the paper actually shows is \exists message M, s.t. \forall decontaminating functions F, F does not decontaminate M. In fact, we know the title (as I interpreted it) is false - the paper gives an example of a message that can be read as plain text which can be truly decontaminated. Basically, when we get a message, we'll be able to immediately tell whether we can decontaminate it or not. If not, the paper argues we're doomed because eventually it'll leak. I just don't find that to be very interesting, and definitely not as strong a claim as the title (as I interpreted it).
- pdonis 9y ago> when we get a message, we'll be able to immediately tell whether we can decontaminate it or not I don't think this is true (at least not "immediately"). The paper is arguing that we will need computers to help decode any non-trivial message, and any computer program capable of decoding a non-trivial message will be Turing complete, and hence capable of running virus code that could have damaging or destructive effects. (The paper's example was a message containing equations in LaTeX, which is Turing complete, hence an ETI could embed a virus in LaTeX code, and humans might not be able to spot it.) The strategy the paper is advocating, as far as I can see, is that, unless the message is simple enough that humans can decode it without computer help, and thereby check directly whether it contains any dangerous information, the message should be destroyed. I suppose this could count as "we can't tell whether we can decontaminate it", but I'm not sure that's what you meant.
- IntronExon 9y agoI suppose one option is to stick a computer, not networked, on a small spacecraft along with a single astronaut and pop them into LEO. Run the code, and have the astronaut tell us what it says or does. If something goes wrong, the whole thing deorbits. If we’re really paranoid we could send it well past LEO. Edit: If we were incredibly paranoid and had some self-sacrificing astronauts it could be a one-way trip to Mercury.