12 ms·

Interstellar communication. IX. Message decontamination is impossible

107 points by mkhalil 9y ago

obblekk 9y ago
I read this paper (it's only a few pages, completely non-technical). The entire paper assumes we get a certain type of communication and in the course of decoding it, destroy humanity. It would make an interesting screen play, but the paper presents no proof whatsoever that "message decontamination is impossible." At best, the paper presents 1 case and outlines 1 possible ending which is the end of humanity. Kinda interesting but not nearly as universal as the title claims.
- k__ 9y ago
  Wasn't Arrival about this?
  - knowaveragejoe 9y ago
    I read both the short story and saw the film, and I didn't get that from it.
  - spartanatreyu 9y ago
    Nope. <spoilers>It's about two things. First, an alien race that give's humanity a "weapon" which is their language. Anyone who learns the language starts to perceive time in a non-linear (probably deterministic) fashion, ie. flashbacks and flashfowards. They did this so that humanity would save them sometime in the future from something unknown to modern day humans. And second, it's about how a translator lives through the life and death of her child while understanding said language. She makes the choice of conceiving her child even though she knew the child was going to die in the future because it was worth it to her because of how much she loved her daughter.</spoliers>
    - e40 9y ago
      That is a very good summary of the movie. I've watched it 3 times and love it more each time.
      - Simon_says 9y ago
        Did you read the short story? IMO, it's better.
        colatkinson 9y ago
        Link to the story, for GP: http://discours.philol.msu.ru/attachments/article/264/Chiang_Story%20of%20Your%20Life.pdf http://discours.philol.msu.ru/attachments/article/264/Chiang...
    - k__ 9y ago
      Yes, that is what happens, but the whole language was like virus that turned people into p-zombies.
  - riffraff 9y ago
    Snow Crash was closer, if anything.
  - MattRix 9y ago
    Not exactly, Arrival is more about how learning a new language can change the way you think. If you want a fiction novel that IS very similar to this, I recommend reading The Three-Body Problem.
    - mirimir 9y ago
      That is good. Earlier, as EvanAnderson notes, there was the Blight in Vernor Vinge's A Fire Upon the Deep. Also the malware attack on the Ceres colony in The Killing Star, by Charles Pellegrino and George Zebrowski.
- wingspar 9y ago
  Sounds like Monty Pythons killer joke... https://m.youtube.com/watch?v=_yo9WHrTvks https://m.youtube.com/watch?v=_yo9WHrTvks https://en.m.wikipedia.org/wiki/The_Funniest_Joke_in_the_World https://en.m.wikipedia.org/wiki/The_Funniest_Joke_in_the_Wor...
  - api 9y ago
    Monty python solved the Fermi paradox!
- shkkmo 9y ago
  > The entire paper assumes we get a certain type of communication and in the course of decoding it destroy humanity. "Assume" is a bit of an odd way of putting it. The paper is a discussion of the risks of receiving a certain type of message. > but the paper presents no proof whatsoever that "message decontamination is impossible." That's a bit harsh. The paper provides several arguments which I find quite solid. Which of them did you find unconvincing? The existence of message that can't be parsed by hand? The risk of manipulative AI running on an air-gapped computer? The difficulty of containing a message given that we can't turn off the source? The difficulty of containing knowledge/information over longer time scales? I didn't think there was much particularly new in that article. It's mostly a re-hashing of other people's arguments. The most interesting point the article made is this conclusion: "As we realize that some message types are potentially dangerous, we can adapt our own peaceful transmissions accordingly. We should certainly not transmit any code. Instead, a plain text encyclopedia (Heidmann 1993), images, music etc. in a simple format are adequate. No advanced computer should be required to decrypt our message."
  - wyattpeak 9y ago
    Personally, I find the idea that "humans could be bargained into a poor position" to be fairly weak. It's certainly possible, but negotiations with hidden agendas have been part and parcel of humanity for all time, and are a major basis for all foreign (and most internal) affairs. The paper presents no predicament that China couldn't put the US in tomorrow with a suitably (apparently) one-sided deal, and yet countries don't as a rule isolate themselves to avoid hearing potentially deadly trade offers.
    - pdonis 9y ago
      > I find the idea that "humans could be bargained into a poor position" to be fairly weak That's because you're assuming that whatever ETI or AI we are bargaining with has roughly our intelligence. That is what has made it possible for humans to negotiate with each other instead of isolating themselves to avoid hearing potentially deadly offers. But any ETI or AI capable of sending us a complex message will probably be much more intelligent than we are. In that case, we would basically be in the same position as, say, a dog trying to negotiate with humans; we simply would not even be able to comprehend what the other side was doing or thinking.
- chicob 9y ago
  ETI message? Kill it with fire. http://tvtropes.org/pmwiki/pmwiki.php/Main/ItsTheOnlyWayToBeSure http://tvtropes.org/pmwiki/pmwiki.php/Main/ItsTheOnlyWayToBe...
  - gmueckl 9y ago
    In the paper, they would nuke the hypothetical moon base, although sadly not from orbit.
- Retra 9y ago
  I came up with an idea as a fantasy world-building exercise which included a confusing book of spells, which if someone where clever enough to decode the central theme of the book, they would have performed the mental gymnastics necessary to cast a spell which would erase every possible future in which they did not become a supreme evil. But here the basic premise is that merely thinking the right thoughts can physically change reality. Which is 'sort-of' true for real brains, but there's an important layer of indirection that makes it not work the way I spelled out above.
- AdamJacobMuller 9y ago
  That was my immediate reaction as well, "decontamination is impossible" um, why? I deal with 'unusual' payloads all the time and I've never seen anything that has this potential, I don't see how anything could be built to be so, without obviously being so. EG: the progression would go from something being some form of plain text, to it being encoded in some way which is very very likely. On the extreme ends say something is sent as machine code. Firstly, this is very odd and suspicious for a lot of reasons but even still you can run untrusted machine code in a very safe way. Assume its resistant to existing types of disassembly for whatever reason. Let's say you want to take an absurd level of paranoia (which, really, is justified if an alien civilization sends us machine code), rut it on an entirely physically disconnected (battery powered) machine inside a Faraday cage. If the code still won't work (ERR: must be connected to the internet to proceed) you can basically assume its malicious. And, after all that, there are far easier ways to kill us all. Send the plans to a massive superweapon but only send it to the USA (or just to Russia, or just to China) but send the other side a message telling them what you did. The snubbed nation may feel the need to make a preemptive attack before the other nation can develop the technology. Send plans for an ultimate bioweapon to the right/wrong people.
  - sbierwagen 9y ago
    >rut it on an entirely physically disconnected (battery powered) machine inside a Faraday cage What if decompression produced an alien message hundreds of gigabytes in length? Is that file completely safe to transmit to un-airgapped computers? Would you bet all of human civilization that it's safe? No way any part of it would produce any negative effect on a human reading it? >Send the plans to a massive superweapon but only send it to the USA Laser beams sent over interstellar distances will be hundreds of thousands of kilometres wide at the destination. The threat model for a hostile probe inside our solar system would be rather different, of course.
    - 13of40 9y ago
      "This is a national security issue that must be treated with the utmost care. Can you analyze and decode this message we received from Alpha Centauri 4?" [Time passes] "By Jove, I think we've cracked it, Sir. It's essentially Huffman encoded and contains a mechanism for repeating redundant strings. Let me see if I can decode the entire message." [Time passes] "Shit, my computer crashed."
- rspeer 9y ago
  It's important to keep in mind that the barrier to entry to publishing on arXiv is that you can produce a well-formed LaTeX document with coherent paragraphs. There's no reason to believe a claim _just_ because it's on arXiV.
  - qubex 9y ago
    Note how they wrote of a ‘proof’ of the famously unproven Riemann Zeta hypothesis and proceeded to merely state it (in a form that might even be compatible with the proven Euler Zeta hypothesis, the ungeneralised case where s is a real number and not a complex number).
alex_young 9y ago
Paper says it's focus is on all ET communications, but instead focuses on malicious AI code. Secure isolation review of code is dismissed because AIs can trick people into freeing them. Finding states: "[M]essage cannot be decontaminated with certainty, and technical risks remain which can pose an existential threat. Complex messages would need to be destroyed in the risk averse case." Couldn't we simply agree not to execute un-audited code from unknown third parties? Seems like the same threat vector as unknown executable code delivered over email to me.
- deleted 9y ago
  [deleted]
- lkrubner 9y ago
  I have something urgent I need to tell you. Your life could be in danger. You need to contact me immediately.
  - IntronExon 9y ago
    Or more effective and sinister, appeal to one person in a posotion to run the code. “Want be rich/immortal/powerful beyond human limits?”
    - perl4ever 9y ago
      Furthermore, that strategy doesn't have to be a one shot deal, where the message says "believe me" before it is proven trustworthy. You can have little pieces of information that demonstrate neat tricks, leading to progressively increasing trust until the full payload is unveiled, for better or worse.
  - lkrubner 9y ago
    I mean, that's an example. There are many ways they could try to trick us. Presumably we are talking about hyper-intelligent beings.
  - biesnecker 9y ago
dsr_ 9y ago
For a glimpse at a future humanity driven to (understandable) paranoia by this scenario, see Ken MacLeod's _The Cassini Division_.
dilyevsky 9y ago
Pretty sure the first message to be received will be something along the lines of "hot dates in your local cloud" or perhaps a plea to send some credits to unlock Andromeda's King hidden bank account.
gene-h 9y ago
It is certainly quite uncommon to see "Clarke and Kubrick" cited in a paper.
21 9y ago
Let's go one step further. Even receiving and recording the message might be very dangerous. There have been cases in the past where contaminated MP3 or video files took over the media players (because of decoding bugs in them). Imagine a very weak signal which requires complicated signal processing and correlation between multiple receivers to reconstruct.
- shkkmo 9y ago
  > There have been cases in the past where contaminated MP3 or video files took over the media players (because of decoding bugs in them). The paper does talk about message compression being a vulnerability since we would have a hard time manually running their decompression algorithm. However, I find it hard to believe that any kind of message will be dangerous due to signal processing without the sender having knowledge about our IT system structure and implementation details or some sort of feedback cycle. Given the time delays of interstellar messaging, I feel the only risk here can come from some sort of AI running locally which I assume would only be possible if we evaluate the message in a turing complete tool (such as their provided decompression algorithm.
  - 21 9y ago
    I get you point, I also thought of this. But at the same time, with long enough messages you could sample a lot of possible IT system structures. The unknown danger here is that there might be some underlying generic exploitable structure in our systems or processes that we are not aware of yet.
    - shkkmo 9y ago
      > The unknown danger here is that there might be some underlying generic exploitable structure in our systems or processes that we are not aware of yet. You would need more than an exploit, you need an exploit that allows you to parse the message in a turing complete fashion to bootstrap an AI. Otherwise, I don't see how identifying the exploit transitions to being a significant risk to humanity.
danidiaz 9y ago
Looks like the plot of "A for Andromeda" https://en.wikipedia.org/wiki/A_for_Andromeda https://en.wikipedia.org/wiki/A_for_Andromeda
- utopkara 9y ago
  Thank you for pointing this out. I heard about this series but never watched it, and didn't know the plot. Amazing how the scifi writers leapfrog their times in understanding of science and its implications.
- oldandtired 9y ago
  Yes, and look at the year of publication. I liked it when I first read it all those decades ago. I still read it occasionally, I especially like the references to the company Intel in the book.
scandox 9y ago
Decoding the messages will probably mine ET-bitcoin for some galactic-scale speculators.
ggm 9y ago
Richard Rhodes' book on the physics and history of science behind the bomb goes to some length to explain just how much information about fission and fusion is a matter of public record. Yet, small details remain redacted. mostly if I recall correctly, about 'levitated cores' in fission bombs, and the exact composition of the 'lens' in a fissile igniter for a fusion bomb. Another source I believe comments that with one infinitesimal change to fundamental universal constants, fission and fusion would be impossible except at stellar scale. Odd, that we live in a universe where the boundary between existence and annihilation is so thin, yet we continue to exist. Personally, I think the proof here is a divide-by-zero instance. Conceptually, a design of information that can be communicated, such that it cannot be comprehended or enacted but its consequence ensues. Isn't that a kind of meme?
- crooked-v 9y ago
  > Odd, that we live in a universe where the boundary between existence and annihilation is so thin, yet we continue to exist. If the universal constants were different, there could well be a different form of life making the comment instead. For example, see Greg Egan's scifi book The Clockwork Rocket, which features a universe that runs on fundamentally different principles but that has broadly recognizable forms of life.
EtDybNuvCu 9y ago
The report cites Bostrom~ But seriously, confined evaluation was figured out in the mid-90s by the E folks, building on years of research into capability security, and the worst thing that can happen is Turing-completeness. I'd be more worried that the message contains a memetic hazard. Our computers wouldn't be infected; our minds would be infected.
- ars 9y ago
  ...if such a thing actually exists.
SCHiM 9y ago
I've been fascinated by these AI containment/usage scenarios ever since I heard of them. Here's another perspective on the problem that I find very fascinating: Assume that, beyond a shadow of a doubt, we have an impenetrable system for interfacing with the AI. I say system, because the people that are part of the protocol are all moral equivalents of Jesus. They can't be corrupted, they can't be blackmailed. The AI is kept in a box that cannot be accessed by anyone else. In short: we've solved the containment problem (of the AI). Of course we _do_ want to use the AI. We've installed it in the box for a reason. If we didn't intend to use it we might as well not have built it. So actually using the tech that it gives us, after checking for corrupting technology, is also a given in this scenario. We can ask the AI questions and interact with it, extract knowledge and have it solve our problems for us. If we were to use the AI, it would still destroy us, even without corrupting any of the people working with it or feeding us corrupted technology. Simply _using_ the AI will be enough. Using the AI effectively cheapens the creation of another AI. Each new processor we have the AI design will be faster and than it's predecessors. Every mathematical problem that it solves is checked, confirmed an shared in our universities. Eventually the technological over-saturation of our society will ensure that the contemporary equivalent of a mobile phone can run a strong AI. The ubiquitousness of the new insights will ensure that all the theory needed to bootstrap another AI is there for the taking. In short, truly _using_ the AI, (asking the questions, sharing the knowledge) will ensure another AI existing outside of the containment system, malicious or otherwise. Taken to the extreme, if strong recursive AI is a possible, it is a given. I don't think strong self improving AI can actually exist. But who knows?
- amptorn 9y ago
  > Using the AI effectively cheapens the creation of another AI. Each new processor we have the AI design will be faster and than it's predecessors. [...] Eventually the technological over-saturation of our society will ensure that the contemporary equivalent of a mobile phone can run a strong AI. These are extremely strong statements, which you cannot prove.
  - SCHiM 9y ago
    Well we do have strong AI in a form factor the size of an average human's head. And that AI, while not super human in most cases, is not specially tuned to be intelligent for the sake of intelligence. It's only as intelligent as evolution 'needed' it to be to survive. I can't prove, but I'm convinced that it is possible. You don't have to be convinced, but I hope you see why I state my case so confidently.
utopkara 9y ago
Fantastic thought experiment. Not sure if it was already covered in a scifi story before, my scifi literature is pretty weak. If not, somebody should turn this into one.
MattRix 9y ago
I just finished reading The Three-Body Problem, which covers some of the same ground... though the aliens in that book end up communicating in some more exotic ways. https://en.wikipedia.org/wiki/The_Three-Body_Problem_(novel) https://en.wikipedia.org/wiki/The_Three-Body_Problem_(novel)
- weerd 9y ago
  Maybe it's been too long since I read this, but I feel like it wasn't adequately explained how the aliens understood the initial message. All of the sudden communication was taking place...
ars 9y ago
They talk as if they have some sort of scientific or mathematical proof of this. But all they have is "it seems to me that". You can publish that, sure, think tanks do that kind of thing all the time. But that's not a scientific paper, not as I understand the concept.
msingle 9y ago
Makes Stanislaw Lem seem even more predictive than usual (https://en.wikipedia.org/wiki/His_Master%27s_Voice_(novel) https://en.wikipedia.org/wiki/His_Master%27s_Voice_(novel))
teolandon 9y ago
Paper is just a wrapper for the AI box problem, nothing new. It doesn't even provide a very strong argument for the impossibility of decontamination, just a "proof" by example (not really a proof).
- ryan-allen 9y ago
  Here's a link to the AI box problem on Wikipedia for anyone like me who hadn't heard of it: https://en.wikipedia.org/wiki/AI_box https://en.wikipedia.org/wiki/AI_box Also, this reminds me of the premise of the movie Ex Machina (no spoilers!) which is sweet! http://www.imdb.com/title/tt0470752/ http://www.imdb.com/title/tt0470752/
drefanzor 9y ago
I'm reading Infinity Born by Douglas E. Richards - it touches on many of the AGI (artificial general intelligence) concepts talked about here. The guy in the book creates an AGI, but thinks he has it contained but it evolves to the point where it gets out of it's box. Good stuff, highly recommend.
nycdotnet 9y ago
E.T. says I just have to enable macros in this interstellar doc to get some Reese's Pieces!
walrus01 9y ago
Here's an interesting thought experiment. If you were going to transmit gigabytes of data to another star system with the intention that a technologically sophisticated society on the other end could decode and reassemble it, what parity and checksumming scheme would you use? For example, treat it like a very poor NNTP service and include 60% PAR2 files? If you were going to use some form of lossless compression, what compressor would you choose that could be simply and quickly explained in a preamble series of data?
gmueckl 9y ago
I cannot decide if this paper is meant to be a joke. Also, there is no proof that true AI can be achieved with Turing completeness alone, so this is a gaping hole in the logic right there.
- sbierwagen 9y ago
  Do you believe the human brain is not a computational system?
  - gmueckl 9y ago
    The human brain does not follow a computer architecture that we can replicate in a lab.
    - sbierwagen 9y ago
      A computer architecture that we can ever replicate in a lab? The human brain is entirely beyond the ability of any computer to simulate it, ever?
      - gmueckl 9y ago
        Should we master replicating that, we also must gain a much better understanding of AI and human intelligence along the way. Our current computers are certainly not able to simulate a human brain. No computer in the world has currently the computing power to run decent, biochemistry based models for each neuron in a human brain. This basically calls for specialized hardware, which may or may not extend the limits of computable problems beyond Turing completeness. If we manage to get to that point, we will have to face more important ethical challenges anyway, like how do we deal with tools that allow us to assess what a person is thinking and planning with near certainty? At that point, the key argument of the paper might actually fall apart because there is a chance that we end up understanding the inner workings of the alien AI. But at that point we have heaped up so much speculation that decent science fiction authors are about to get jealous.
sllabres 9y ago
If it's not possible it is probably the reason for the great filter [1] :-) [1] https://en.wikipedia.org/wiki/Great_Filter https://en.wikipedia.org/wiki/Great_Filter
rumcajz 9y ago
It's worth reading Stanislaw Lem's His Master's Voice. It deals exactly with this topic but it is much more nuanced.
zdw 9y ago
Reminds me of the episode of ST:TNG where they come upon a Borg that was injured, and debate whether to show him a puzzle virus that would cause the entire Borg collective to grind to a halt by decoding it: http://memory-alpha.wikia.com/wiki/I_Borg_(episode) http://memory-alpha.wikia.com/wiki/I_Borg_(episode)
qubex 9y ago
What a load of speculative, thin twaddle. This is just an extension of the never-ending “hard takeoff” malicious-AI argument (as exemplified by “Our Final Invention” by James Barrat and others). “It might cause a mob” or “it might contain a compressed AI that ingratiates itself and betrays us” is really not a terribly convincing argument. It's just... paranoid and meek.
thesz 9y ago
I MUST refer to "Белая трость калибра 7,65" ("White cane of 7.65 caliber"): http://lib.ru/SOCFANT/NEFF/trost.txt http://lib.ru/SOCFANT/NEFF/trost.txt (Russian translation of Czech SciFi piece) I read it in "Tech for Youngs" USSR magazine (Техника Молодёжи) a long time ago, around 1980. It is short and profound, for example, I later came around human echolocation phenomena and instantly remembered that piece. Our current human echolocators are about as good as the main hero of the piece above. Given that, I think we are safe. ;)

107 points by mkhalil 9y ago

I read this paper (it's only a few pages, completely non-technical). The entire paper assumes we get a certain type of communication and in the course of decoding it, destroy humanity. It would make an interesting screen play, but the paper presents no proof whatsoever that "message decontamination is impossible." At best, the paper presents 1 case and outlines 1 possible ending which is the end of humanity. Kinda interesting but not nearly as universal as the title claims.

k__ 9y ago
Wasn't Arrival about this?
- knowaveragejoe 9y ago
  I read both the short story and saw the film, and I didn't get that from it.
- spartanatreyu 9y ago
  Nope. <spoilers>It's about two things. First, an alien race that give's humanity a "weapon" which is their language. Anyone who learns the language starts to perceive time in a non-linear (probably deterministic) fashion, ie. flashbacks and flashfowards. They did this so that humanity would save them sometime in the future from something unknown to modern day humans. And second, it's about how a translator lives through the life and death of her child while understanding said language. She makes the choice of conceiving her child even though she knew the child was going to die in the future because it was worth it to her because of how much she loved her daughter.</spoliers>
  - e40 9y ago
    That is a very good summary of the movie. I've watched it 3 times and love it more each time.
    - Simon_says 9y ago
      Did you read the short story? IMO, it's better.
      - colatkinson 9y ago
        Link to the story, for GP: http://discours.philol.msu.ru/attachments/article/264/Chiang_Story%20of%20Your%20Life.pdf http://discours.philol.msu.ru/attachments/article/264/Chiang...
  - k__ 9y ago
    Yes, that is what happens, but the whole language was like virus that turned people into p-zombies.
- riffraff 9y ago
  Snow Crash was closer, if anything.
- MattRix 9y ago
  Not exactly, Arrival is more about how learning a new language can change the way you think. If you want a fiction novel that IS very similar to this, I recommend reading The Three-Body Problem.
  - mirimir 9y ago
    That is good. Earlier, as EvanAnderson notes, there was the Blight in Vernor Vinge's A Fire Upon the Deep. Also the malware attack on the Ceres colony in The Killing Star, by Charles Pellegrino and George Zebrowski.
wingspar 9y ago
Sounds like Monty Pythons killer joke... https://m.youtube.com/watch?v=_yo9WHrTvks https://m.youtube.com/watch?v=_yo9WHrTvks https://en.m.wikipedia.org/wiki/The_Funniest_Joke_in_the_World https://en.m.wikipedia.org/wiki/The_Funniest_Joke_in_the_Wor...
- api 9y ago
  Monty python solved the Fermi paradox!
shkkmo 9y ago
> The entire paper assumes we get a certain type of communication and in the course of decoding it destroy humanity. "Assume" is a bit of an odd way of putting it. The paper is a discussion of the risks of receiving a certain type of message. > but the paper presents no proof whatsoever that "message decontamination is impossible." That's a bit harsh. The paper provides several arguments which I find quite solid. Which of them did you find unconvincing? The existence of message that can't be parsed by hand? The risk of manipulative AI running on an air-gapped computer? The difficulty of containing a message given that we can't turn off the source? The difficulty of containing knowledge/information over longer time scales? I didn't think there was much particularly new in that article. It's mostly a re-hashing of other people's arguments. The most interesting point the article made is this conclusion: "As we realize that some message types are potentially dangerous, we can adapt our own peaceful transmissions accordingly. We should certainly not transmit any code. Instead, a plain text encyclopedia (Heidmann 1993), images, music etc. in a simple format are adequate. No advanced computer should be required to decrypt our message."
- wyattpeak 9y ago
  Personally, I find the idea that "humans could be bargained into a poor position" to be fairly weak. It's certainly possible, but negotiations with hidden agendas have been part and parcel of humanity for all time, and are a major basis for all foreign (and most internal) affairs. The paper presents no predicament that China couldn't put the US in tomorrow with a suitably (apparently) one-sided deal, and yet countries don't as a rule isolate themselves to avoid hearing potentially deadly trade offers.
  - pdonis 9y ago
    > I find the idea that "humans could be bargained into a poor position" to be fairly weak That's because you're assuming that whatever ETI or AI we are bargaining with has roughly our intelligence. That is what has made it possible for humans to negotiate with each other instead of isolating themselves to avoid hearing potentially deadly offers. But any ETI or AI capable of sending us a complex message will probably be much more intelligent than we are. In that case, we would basically be in the same position as, say, a dog trying to negotiate with humans; we simply would not even be able to comprehend what the other side was doing or thinking.
chicob 9y ago
ETI message? Kill it with fire. http://tvtropes.org/pmwiki/pmwiki.php/Main/ItsTheOnlyWayToBeSure http://tvtropes.org/pmwiki/pmwiki.php/Main/ItsTheOnlyWayToBe...
- gmueckl 9y ago
  In the paper, they would nuke the hypothetical moon base, although sadly not from orbit.
Retra 9y ago
I came up with an idea as a fantasy world-building exercise which included a confusing book of spells, which if someone where clever enough to decode the central theme of the book, they would have performed the mental gymnastics necessary to cast a spell which would erase every possible future in which they did not become a supreme evil. But here the basic premise is that merely thinking the right thoughts can physically change reality. Which is 'sort-of' true for real brains, but there's an important layer of indirection that makes it not work the way I spelled out above.
AdamJacobMuller 9y ago
That was my immediate reaction as well, "decontamination is impossible" um, why? I deal with 'unusual' payloads all the time and I've never seen anything that has this potential, I don't see how anything could be built to be so, without obviously being so. EG: the progression would go from something being some form of plain text, to it being encoded in some way which is very very likely. On the extreme ends say something is sent as machine code. Firstly, this is very odd and suspicious for a lot of reasons but even still you can run untrusted machine code in a very safe way. Assume its resistant to existing types of disassembly for whatever reason. Let's say you want to take an absurd level of paranoia (which, really, is justified if an alien civilization sends us machine code), rut it on an entirely physically disconnected (battery powered) machine inside a Faraday cage. If the code still won't work (ERR: must be connected to the internet to proceed) you can basically assume its malicious. And, after all that, there are far easier ways to kill us all. Send the plans to a massive superweapon but only send it to the USA (or just to Russia, or just to China) but send the other side a message telling them what you did. The snubbed nation may feel the need to make a preemptive attack before the other nation can develop the technology. Send plans for an ultimate bioweapon to the right/wrong people.
- sbierwagen 9y ago
  >rut it on an entirely physically disconnected (battery powered) machine inside a Faraday cage What if decompression produced an alien message hundreds of gigabytes in length? Is that file completely safe to transmit to un-airgapped computers? Would you bet all of human civilization that it's safe? No way any part of it would produce any negative effect on a human reading it? >Send the plans to a massive superweapon but only send it to the USA Laser beams sent over interstellar distances will be hundreds of thousands of kilometres wide at the destination. The threat model for a hostile probe inside our solar system would be rather different, of course.
  - 13of40 9y ago
    "This is a national security issue that must be treated with the utmost care. Can you analyze and decode this message we received from Alpha Centauri 4?" [Time passes] "By Jove, I think we've cracked it, Sir. It's essentially Huffman encoded and contains a mechanism for repeating redundant strings. Let me see if I can decode the entire message." [Time passes] "Shit, my computer crashed."
rspeer 9y ago
It's important to keep in mind that the barrier to entry to publishing on arXiv is that you can produce a well-formed LaTeX document with coherent paragraphs. There's no reason to believe a claim _just_ because it's on arXiV.
- qubex 9y ago
  Note how they wrote of a ‘proof’ of the famously unproven Riemann Zeta hypothesis and proceeded to merely state it (in a form that might even be compatible with the proven Euler Zeta hypothesis, the ungeneralised case where s is a real number and not a complex number).

alex_young 9y ago

Paper says it's focus is on all ET communications, but instead focuses on malicious AI code. Secure isolation review of code is dismissed because AIs can trick people into freeing them. Finding states: "[M]essage cannot be decontaminated with certainty, and technical risks remain which can pose an existential threat. Complex messages would need to be destroyed in the risk averse case." Couldn't we simply agree not to execute un-audited code from unknown third parties? Seems like the same threat vector as unknown executable code delivered over email to me.

deleted 9y ago
[deleted]
lkrubner 9y ago
I have something urgent I need to tell you. Your life could be in danger. You need to contact me immediately.
- IntronExon 9y ago
  Or more effective and sinister, appeal to one person in a posotion to run the code. “Want be rich/immortal/powerful beyond human limits?”
  - perl4ever 9y ago
    Furthermore, that strategy doesn't have to be a one shot deal, where the message says "believe me" before it is proven trustworthy. You can have little pieces of information that demonstrate neat tricks, leading to progressively increasing trust until the full payload is unveiled, for better or worse.
- lkrubner 9y ago
  I mean, that's an example. There are many ways they could try to trick us. Presumably we are talking about hyper-intelligent beings.
- biesnecker 9y ago

dsr_ 9y ago

For a glimpse at a future humanity driven to (understandable) paranoia by this scenario, see Ken MacLeod's _The Cassini Division_.

dilyevsky 9y ago

Pretty sure the first message to be received will be something along the lines of "hot dates in your local cloud" or perhaps a plea to send some credits to unlock Andromeda's King hidden bank account.

gene-h 9y ago

It is certainly quite uncommon to see "Clarke and Kubrick" cited in a paper.

21 9y ago

Let's go one step further. Even receiving and recording the message might be very dangerous. There have been cases in the past where contaminated MP3 or video files took over the media players (because of decoding bugs in them). Imagine a very weak signal which requires complicated signal processing and correlation between multiple receivers to reconstruct.

shkkmo 9y ago
> There have been cases in the past where contaminated MP3 or video files took over the media players (because of decoding bugs in them). The paper does talk about message compression being a vulnerability since we would have a hard time manually running their decompression algorithm. However, I find it hard to believe that any kind of message will be dangerous due to signal processing without the sender having knowledge about our IT system structure and implementation details or some sort of feedback cycle. Given the time delays of interstellar messaging, I feel the only risk here can come from some sort of AI running locally which I assume would only be possible if we evaluate the message in a turing complete tool (such as their provided decompression algorithm.
- 21 9y ago
  I get you point, I also thought of this. But at the same time, with long enough messages you could sample a lot of possible IT system structures. The unknown danger here is that there might be some underlying generic exploitable structure in our systems or processes that we are not aware of yet.
  - shkkmo 9y ago
    > The unknown danger here is that there might be some underlying generic exploitable structure in our systems or processes that we are not aware of yet. You would need more than an exploit, you need an exploit that allows you to parse the message in a turing complete fashion to bootstrap an AI. Otherwise, I don't see how identifying the exploit transitions to being a significant risk to humanity.

danidiaz 9y ago

Looks like the plot of "A for Andromeda" https://en.wikipedia.org/wiki/A_for_Andromeda https://en.wikipedia.org/wiki/A_for_Andromeda

utopkara 9y ago
Thank you for pointing this out. I heard about this series but never watched it, and didn't know the plot. Amazing how the scifi writers leapfrog their times in understanding of science and its implications.
oldandtired 9y ago
Yes, and look at the year of publication. I liked it when I first read it all those decades ago. I still read it occasionally, I especially like the references to the company Intel in the book.

scandox 9y ago

Decoding the messages will probably mine ET-bitcoin for some galactic-scale speculators.

ggm 9y ago

Richard Rhodes' book on the physics and history of science behind the bomb goes to some length to explain just how much information about fission and fusion is a matter of public record. Yet, small details remain redacted. mostly if I recall correctly, about 'levitated cores' in fission bombs, and the exact composition of the 'lens' in a fissile igniter for a fusion bomb. Another source I believe comments that with one infinitesimal change to fundamental universal constants, fission and fusion would be impossible except at stellar scale. Odd, that we live in a universe where the boundary between existence and annihilation is so thin, yet we continue to exist. Personally, I think the proof here is a divide-by-zero instance. Conceptually, a design of information that can be communicated, such that it cannot be comprehended or enacted but its consequence ensues. Isn't that a kind of meme?

crooked-v 9y ago
> Odd, that we live in a universe where the boundary between existence and annihilation is so thin, yet we continue to exist. If the universal constants were different, there could well be a different form of life making the comment instead. For example, see Greg Egan's scifi book The Clockwork Rocket, which features a universe that runs on fundamentally different principles but that has broadly recognizable forms of life.

EtDybNuvCu 9y ago

The report cites Bostrom~ But seriously, confined evaluation was figured out in the mid-90s by the E folks, building on years of research into capability security, and the worst thing that can happen is Turing-completeness. I'd be more worried that the message contains a memetic hazard. Our computers wouldn't be infected; our minds would be infected.

ars 9y ago
...if such a thing actually exists.

SCHiM 9y ago

I've been fascinated by these AI containment/usage scenarios ever since I heard of them. Here's another perspective on the problem that I find very fascinating: Assume that, beyond a shadow of a doubt, we have an impenetrable system for interfacing with the AI. I say system, because the people that are part of the protocol are all moral equivalents of Jesus. They can't be corrupted, they can't be blackmailed. The AI is kept in a box that cannot be accessed by anyone else. In short: we've solved the containment problem (of the AI). Of course we _do_ want to use the AI. We've installed it in the box for a reason. If we didn't intend to use it we might as well not have built it. So actually using the tech that it gives us, after checking for corrupting technology, is also a given in this scenario. We can ask the AI questions and interact with it, extract knowledge and have it solve our problems for us. If we were to use the AI, it would still destroy us, even without corrupting any of the people working with it or feeding us corrupted technology. Simply _using_ the AI will be enough. Using the AI effectively cheapens the creation of another AI. Each new processor we have the AI design will be faster and than it's predecessors. Every mathematical problem that it solves is checked, confirmed an shared in our universities. Eventually the technological over-saturation of our society will ensure that the contemporary equivalent of a mobile phone can run a strong AI. The ubiquitousness of the new insights will ensure that all the theory needed to bootstrap another AI is there for the taking. In short, truly _using_ the AI, (asking the questions, sharing the knowledge) will ensure another AI existing outside of the containment system, malicious or otherwise. Taken to the extreme, if strong recursive AI is a possible, it is a given. I don't think strong self improving AI can actually exist. But who knows?

amptorn 9y ago
> Using the AI effectively cheapens the creation of another AI. Each new processor we have the AI design will be faster and than it's predecessors. [...] Eventually the technological over-saturation of our society will ensure that the contemporary equivalent of a mobile phone can run a strong AI. These are extremely strong statements, which you cannot prove.
- SCHiM 9y ago
  Well we do have strong AI in a form factor the size of an average human's head. And that AI, while not super human in most cases, is not specially tuned to be intelligent for the sake of intelligence. It's only as intelligent as evolution 'needed' it to be to survive. I can't prove, but I'm convinced that it is possible. You don't have to be convinced, but I hope you see why I state my case so confidently.

utopkara 9y ago

Fantastic thought experiment. Not sure if it was already covered in a scifi story before, my scifi literature is pretty weak. If not, somebody should turn this into one.

MattRix 9y ago

I just finished reading The Three-Body Problem, which covers some of the same ground... though the aliens in that book end up communicating in some more exotic ways. https://en.wikipedia.org/wiki/The_Three-Body_Problem_(novel) https://en.wikipedia.org/wiki/The_Three-Body_Problem_(novel)

weerd 9y ago
Maybe it's been too long since I read this, but I feel like it wasn't adequately explained how the aliens understood the initial message. All of the sudden communication was taking place...

ars 9y ago

They talk as if they have some sort of scientific or mathematical proof of this. But all they have is "it seems to me that". You can publish that, sure, think tanks do that kind of thing all the time. But that's not a scientific paper, not as I understand the concept.

msingle 9y ago

Makes Stanislaw Lem seem even more predictive than usual (https://en.wikipedia.org/wiki/His_Master%27s_Voice_(novel) https://en.wikipedia.org/wiki/His_Master%27s_Voice_(novel))

teolandon 9y ago

Paper is just a wrapper for the AI box problem, nothing new. It doesn't even provide a very strong argument for the impossibility of decontamination, just a "proof" by example (not really a proof).

ryan-allen 9y ago
Here's a link to the AI box problem on Wikipedia for anyone like me who hadn't heard of it: https://en.wikipedia.org/wiki/AI_box https://en.wikipedia.org/wiki/AI_box Also, this reminds me of the premise of the movie Ex Machina (no spoilers!) which is sweet! http://www.imdb.com/title/tt0470752/ http://www.imdb.com/title/tt0470752/

drefanzor 9y ago

I'm reading Infinity Born by Douglas E. Richards - it touches on many of the AGI (artificial general intelligence) concepts talked about here. The guy in the book creates an AGI, but thinks he has it contained but it evolves to the point where it gets out of it's box. Good stuff, highly recommend.

nycdotnet 9y ago

E.T. says I just have to enable macros in this interstellar doc to get some Reese's Pieces!

walrus01 9y ago

Here's an interesting thought experiment. If you were going to transmit gigabytes of data to another star system with the intention that a technologically sophisticated society on the other end could decode and reassemble it, what parity and checksumming scheme would you use? For example, treat it like a very poor NNTP service and include 60% PAR2 files? If you were going to use some form of lossless compression, what compressor would you choose that could be simply and quickly explained in a preamble series of data?

gmueckl 9y ago

I cannot decide if this paper is meant to be a joke. Also, there is no proof that true AI can be achieved with Turing completeness alone, so this is a gaping hole in the logic right there.

sbierwagen 9y ago
Do you believe the human brain is not a computational system?
- gmueckl 9y ago
  The human brain does not follow a computer architecture that we can replicate in a lab.
  - sbierwagen 9y ago
    A computer architecture that we can ever replicate in a lab? The human brain is entirely beyond the ability of any computer to simulate it, ever?
    - gmueckl 9y ago
      Should we master replicating that, we also must gain a much better understanding of AI and human intelligence along the way. Our current computers are certainly not able to simulate a human brain. No computer in the world has currently the computing power to run decent, biochemistry based models for each neuron in a human brain. This basically calls for specialized hardware, which may or may not extend the limits of computable problems beyond Turing completeness. If we manage to get to that point, we will have to face more important ethical challenges anyway, like how do we deal with tools that allow us to assess what a person is thinking and planning with near certainty? At that point, the key argument of the paper might actually fall apart because there is a chance that we end up understanding the inner workings of the alien AI. But at that point we have heaped up so much speculation that decent science fiction authors are about to get jealous.

sllabres 9y ago

If it's not possible it is probably the reason for the great filter [1] :-) [1] https://en.wikipedia.org/wiki/Great_Filter https://en.wikipedia.org/wiki/Great_Filter

rumcajz 9y ago

It's worth reading Stanislaw Lem's His Master's Voice. It deals exactly with this topic but it is much more nuanced.

zdw 9y ago

Reminds me of the episode of ST:TNG where they come upon a Borg that was injured, and debate whether to show him a puzzle virus that would cause the entire Borg collective to grind to a halt by decoding it: http://memory-alpha.wikia.com/wiki/I_Borg_(episode) http://memory-alpha.wikia.com/wiki/I_Borg_(episode)

qubex 9y ago

What a load of speculative, thin twaddle. This is just an extension of the never-ending “hard takeoff” malicious-AI argument (as exemplified by “Our Final Invention” by James Barrat and others). “It might cause a mob” or “it might contain a compressed AI that ingratiates itself and betrays us” is really not a terribly convincing argument. It's just... paranoid and meek.

thesz 9y ago

I MUST refer to "Белая трость калибра 7,65" ("White cane of 7.65 caliber"): http://lib.ru/SOCFANT/NEFF/trost.txt http://lib.ru/SOCFANT/NEFF/trost.txt (Russian translation of Czech SciFi piece) I read it in "Tech for Youngs" USSR magazine (Техника Молодёжи) a long time ago, around 1980. It is short and profound, for example, I later came around human echolocation phenomena and instantly remembered that piece. Our current human echolocators are about as good as the main hero of the piece above. Given that, I think we are safe. ;)