6 ms·
> Grey goo. The Borg. Skynet. ... You're reasoning from fictional evidence, like the people who wanted to ban heart and kidney transplants because they'd watch
by kragen 5y ago
> Grey goo. The Borg. Skynet. ...
You're reasoning from fictional evidence, like the people who wanted to ban heart and kidney transplants because they'd watched Frankenstein, or who feared computers because they'd watched 2001.
Self-reproducing machines are dangerous in the same way that ammonium perchlorate, natural gas, electricity, or computers are dangerous: people can use them to do bad things, and people can engineer systems carelessly, causing accidents.
https://en.wikipedia.org/wiki/PEPCON_disaster https://en.wikipedia.org/wiki/PEPCON_disaster
https://en.wikipedia.org/wiki/New_London_School_explosion https://en.wikipedia.org/wiki/New_London_School_explosion
https://en.wikipedia.org/wiki/War_of_the_currents#Safety_concerns https://en.wikipedia.org/wiki/War_of_the_currents#Safety_con...
https://electricenergyonline.com/energy/magazine/362/article/Prevention-of-Oil-Filled-Transformer-Explosions.htm https://electricenergyonline.com/energy/magazine/362/article...
https://en.wikipedia.org/wiki/Morris_worm https://en.wikipedia.org/wiki/Morris_worm
What is called for is adequate respect for the capabilities and risks of the technology you're working with, prudence in preventing accidents, and avoidance of extreme technological power imbalances like those that led to the Congo Free State and the Google Play Store, not the sort of puerile irrationality that privileges Gene Roddenberry's ideas over those of John von Neumann.
- throwaway316943 5y agoHere’s one from reality, bacteria. If we create self replicating machines, even if we do not intentionally introduce variation, they will vary from generation to generation. They will presumably not be indestructible and so some form of selection will be possible. If their reproduction cycle is fast enough they could potentially deviate from their intended purpose in a very short time.
- Kim_Bruning 5y agoIn fact, all living things form a category of self replicating machines. Depending, your [highly optimized] self-replicator might have some stiff competition.
- danielheath 5y agoThe idea that a designed machine could be competitive with living things at self-replication (the sole target of a multi-millenia optimisation process) seems… pretty unlikely to me.
- throwaway316943 5y agoIt doesn’t seem far fetched to me, evolution cannot see nor understand mistakes it makes. There is still plenty of room for optimization.
- drran 5y agoRemote inspection and checksum of the device firmware can solve this problem. You should worry about intentional hacking of the firmware, i.e. self-replicating botnet.
- nicoburns 5y agoThis assumes that the firmware contains no bugs. As soon as it does, there is always the possibility of unexpected behaviour and an unintended evolutionary trajectory.
- kragen 5y agoI used to be concerned about this issue, due to the obvious analogy with biological evolution, but later I realized that it's an easy risk to guard against. It's a straightforward engineering problem to reduce the possibility of accidental program mutation to any arbitrarily low level, for example using SHA-2 (with which the chance of an undetected error is 1e-77). Of course, you can have "somatic mutations" where one part of a machine or another malfunctions, for example due to damage or errors during construction; but those don't get propagated to the next generation, so they don't produce the kind of progressive deviation you're describing. This doesn't happen in nature for a variety of reasons, among which is that as such corrective mechanisms become progressively more perfect, the evolution of the species using them becomes progressively slower, and therefore the perfection of the error-correction mechanisms never quite arrives. Moreover, any species whose evolution becomes very slow is at a major disadvantage when the environment changes sufficiently; it will probably die out and leave no descendants after the next climate change, even something minor like an Ice Age, much less a meteor strike. Another thing to keep in mind is that the number of generations is quite limited in practice. If, to take an unrealistically risky example, you have an alcohol-dependent nanobot replicator weighing 2 picograms, and you set it to reshaping a tonne of alcohol-soaked soil, it can't make more than 5e17 copies of itself, for which it only needs 59 generations. We aren't talking about thousands or millions of generations: even if time allows for them, space doesn't. 1e-77 (.000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 01) is a number that it can be difficult to get a handle on. If we, to take another unrealistically risky example, converted Ceres into 2-picogram nanobot replicators, there would only be 4.7e35 of them, so the chance that one of them would have a mutation in its program that SHA-2 couldn't detect would be 4e-42, assuming that there were just as many erroneous replications as correct ones. (If only one out of every 1000 nanobot firmware installations had a copying error, the chance that one of them went undetected would instead be 4e-45.) The universe is only 4.3e26 nanoseconds old. So, human malice or extreme carelessness would be necessary.
- phscguy 5y ago>not the sort of puerile irrationality that privileges Gene Roddenberry's ideas over those of John von Neumann. Ah yes, the von Neumann that helped developed both the fission bomb and the hydrogen bomb and aggressively promoted the use of them and tried to start nuclear war. I know von Neumann is held in high regard with respect to his role in computing, but he is hardly a guy worthy of respect for his views on the safety of technology.
- JohnPrine 5y agoMy understanding is that von Neumann recognized that science would produce these types of weapons eventually, and so pushed hard to reach an equilibrium state of non-use of nuclear weapons under Mutually Assured Destruction (the dude did found modern game theory). So I think it's a bit disingenuous to say that he tried to start nuclear war.
- phscguy 5y agoHe advocated for a first strike attack (bombing Moscow). Though this was before the Soviets had a strong nuclear arsenal and so likely would not have started a war. So yeah you are right, saying he was pushing for a war is stretching it. He did push for nuking a major city though.
- kragen 5y agoCertainly von Neumann is far from infallible, but his views on the safety of technology were based on reasoning and knowledge, not what would make exciting prime-time TV. We're literally contrasting the inventor of game theory and modern meteorology* with a TV show where sounds travel through outer space and all the extraterrestrials speak English. This is like discussing whether Stephen Jay Gould or Jimmy Swaggart has a more credible opinion about evolution. I mean, punctuated equilibrium might or might not be correct, but you're being ridiculous. https://whatever.scalzi.com/2009/10/13/teching-the-tech/ https://whatever.scalzi.com/2009/10/13/teching-the-tech/ https://archive.is/khTlp https://archive.is/khTlp > when you admit that Star Trek has as much to do with plausibly extrapolated science as The A-Team has to do with a realistic look at the lives of military veterans, life gets easier. ... Meta to this is the discussion of why we have to accept that film/tv SF is riding the shortbus — there’s no actual reason it has to be that way — but let’s not get into that right at the moment. ______ * As well as, as you point out, a significant contributor to the development of the fission bomb, the hydrogen bomb, and modern computation. He was also the guy who axiomatized quantum mechanics and the foundations of mathematics, discovered continuous geometry and quantum logic and Hilbert spaces, and solved the compact-groups case of Hilbert's fifth problem. I guess you don't know much about mathematics, so that won't mean much to you; suffice it to say that the "von Neumann machine" was among the least of his achievements. Not bad for a chemical engineer.
- sandworm101 5y agoWhen facing new technology, fiction is our only guide. We would be fools to disregard the thinking of great minds about the potential, even likely, consequences of new capabilities. Such is the power of modern technology that we must set guidelines on things that do not yet exist, to prevent them from existing. We don't forbid research into genetic weapons because we have experience with them. We ban such things because, in our imagination, we see their potential.
- kragen 5y agoWe would be fools to disregard the thinking of great minds like John von Neumann, but even more fools to consider Gene Roddenberry a great mind. For the purposes of prognostication, he wasn't a mind at all; he wasn't even trying to predict what would happen in the future, but to dress up 19th-century Horatio Hornblower naval drama and 20th-century social commentary in whizbang clothing, so people would watch it and he could sell Chevrolets and vitamin pills. See https://news.ycombinator.com/item?id=28519184 https://news.ycombinator.com/item?id=28519184.
- playguardin 5y agoI know right. Its not like in real life a virus nerds experiment would go horribly wrong escape the labe and kill millions worldwide just like in a movie.
- marcosdumay 5y agoGrey goo is not so much fiction as it is the logic consequence of the kind of machine its creator "designed" (on the sci-fi definition of designing, but with a very strong focus on science). If we ever build unsupervised self-replicating machines, we will get grey-goo like problems, that is a completely different threat model from any machine we have ever seen. And if the machines are any fast or any far from us, logic dictates that none of the normal safety measures we apply elsewhere will apply. (So, I guess the lesson is "don't build unsupervised self-replicating machines". Even our body limits unsupervised self-replication by limiting how many generations a cell can create. But it is surely not "people can misuse any tool", "use it wisely" or "control the consequences".)
- kragen 5y ago> Grey goo is not so much fiction as it is the logic consequence of the kind of machine its creator "designed" (on the sci-fi definition of designing, but with a very strong focus on science). That's true; I think it's one of the reasons Zyvex's designs didn't include nanobot-sized constructors, instead preferring "convergent assembly" using nanometer-sized parts but not nanometer-sized replicators. Instead, they envisioned that the minimal self-replicating unit would be the size of a laser printer. A person armed with a rock could disable a warehouse full of such devices in a few hours if that became necessary. So, self-replicating nanotechnology doesn't imply a risk of gray goo. As a more useful point of reference than space opera and Drexler's thought-provoking speculations, Linux and GCC are capable of replicating themselves, but they don't pose the same risks as the Morris worm. > that is a completely different threat model from any machine we have ever seen. Even without the bogus gray-goo threat, that's true of self-replicating machines, but so were my other examples at the time. Well, maybe the PEPCON disaster wasn't so different from the items in https://en.wikipedia.org/wiki/List_of_ammonium_nitrate_disasters https://en.wikipedia.org/wiki/List_of_ammonium_nitrate_disas... and https://en.wikipedia.org/wiki/Largest_artificial_non-nuclear_explosions https://en.wikipedia.org/wiki/Largest_artificial_non-nuclear..., in particular https://en.wikipedia.org/wiki/Wanggongchang_Explosion https://en.wikipedia.org/wiki/Wanggongchang_Explosion in 01626, which arguably brought about the end of the Ming 18 years later. Nothing like the Wanggongchang explosion had ever happened before, though, and nothing like it would happen again until the 20th century; quite possibly it was the largest human-made explosion before the US started the first nuclear war in 01945, though there are a couple of other candidates.