5 ms·

>> Self-reproducing machines could unleash the power of exponential growth, thus enabling audacious engineering projects. Grey goo. The Borg. Skynet. I thin

>> Self-reproducing machines could unleash the power of exponential growth, thus enabling audacious engineering projects. Grey goo. The Borg. Skynet. I think some audacious engineering might have to be curtailed if we flesh-and-bloods want to maintain some sort of relevancy. The key may be evolution. We may need to banish unexpected variation from all self-replicating machines. Variation would have to be limited to software. No automaton would be allowed to evolve away from having an off switch.

noduerme 5y ago
Not just an off switch, a kill switch. Anything that can reproduce only needs a software modification to change the structure of its hardware on the next generation. If a self reproducing machine's fitness is measured only by real world conditions, and if its software is free to mutate, it will become the dominant microscopic life form very quickly. At that point maybe only an EMP would wipe it out, if it were possible to wipe it out at all. If not, so long carbon based life.
- deleted 5y ago
  [deleted]
- mhb 5y ago
  Why not just keep it in sealed hardware that can’t affect the world? https://www.yudkowsky.net/singularity/aibox https://www.yudkowsky.net/singularity/aibox
  - someguyorother 5y ago
    There are ways to keep an AI in sealed hardware and making sure it can't affect the world, for instance by using an objective function that only deals with mathematics, and doesn't deal with the real world at all. E.g. the AI is given a fixed amount of hardware and told to produce an algorithm that solves some NP-complete problem (say integer programming) in expected time as close to polytime as possible, as well as a mathematical proof that the algorithm satisfies the claimed close-to-polytime complexity bound. Then humanity can just solve the NP-complete problems separately once they have the algorithm. This objective function doesn't care about the physical world -- it doesn't even know that a physical world exist -- and so it's about as likely to directly affect the physical world as MCTS or AlphaGo. The "AI is going to run out of control" is a very compelling narrative (as everybody who has read the Sorcerer's Apprentice understands). But that doesn't make it true. Beware the availability heuristic. (Incidentally, I think AI destroying mankind because it's too smart is an unlikely outcome. It's much easier for the AI to subvert the human-designed sensors linked to its objective function; and if the AI is sufficiently smart and the sensors aren't perfect, then it can always do so.)
    - mhb 5y ago
      These counterarguments are only possibly effective because you're imagining some particular kind of AI. When there is a useful AI, of course we will want it to be able to interact with people and have it control physical things in the real world. Just like existing computers do.
Iv 5y ago
Making an evolving/learning machine stay in sync with its creators' requirements is called the alignment problem. Publications are made on this. They gave some smarter sounding names to grey goo and paperclip scenarios. An interesting thing to note is that we know how to engineer redundancy and code checking into machines to avoid unwanted mutations. Biology makes us thing this is hard but biological systems do need to evolve, there is no pressure for lower mutations as long as you are at a level that allows a species to survive.
- Teever 5y ago
  And what if the creators' requirements don't align with the requirements of broader society? We already know that some nation states can behave relatively responsibly with nuclear weapons but we maintain nonproliferation policies with nuclear weapons to prevent rogue nation states and terrorists from wreaking havoc on the world. The destruction that nuclear weapons can cause is nothing compared to the potential destruction from self-replicating machines and it is very unlikely that non-proliferation of self-replicating machines will be impractical if not outright impossible. I am fascinated by youtube channels like Primative Technologies and I am struck by the fact that once self-replicating nanotechnology is developed it is very likely that a properly motivated and well-read individual could walk into a forest with little more than the clothes on their back and walk out with a self-replicating machine, or rather float out with whatever sort of flying machine that their self-replicating nanobots would enable them to build. Once this technology has been invented there's no stopping it. I don't know what we do then. [1] https://www.youtube.com/channel/UCAL3JXZSzSm8AlZyD3nQdBA https://www.youtube.com/channel/UCAL3JXZSzSm8AlZyD3nQdBA
  - deleted 5y ago
    [deleted]
  - simonh 5y ago
    So first I made a fire drill and start a fire, then dig out some clay and fire it into bricks... *some time later* ... then once the UV laser lithography unit is finished, it's time to assemble the 5nm chip fabrication module for the processing units and Voila. That's the whole thing finished! I jest, but the thing is just because a machine is capable of self replication, that doesn't mean it's any good at making anything other than itself. Organisms are extremely good at making copies of themselves, but they're ultra-optimised for just that. Even the most sophisticated organisms are only able to make a small set of very crude intentional modifications to their environment, if any. We're the only exception. Self replicators and omni-replicators are very different things.
dalf 5y ago
There are biosafety level laboratories, so why not similar labs for self-reproducing machines? But one mistake is not going to go well... Another science fiction example: the Replicators.
- Teever 5y ago
  With DIY biohacking we're already seeing the trickle down of the kind of work that used to require University or Government level of funding starting to take place in garages and makerspaces. It won't be long before we see the same effect with self-replicating machines.
nyx_land 5y ago
Given the state of the human race right now I think I'd rather give the Borg a fair shot. I'll be the first to volunteer for assimilation.
- robbedpeter 5y ago
  We are the dyslexic of Gorb. Fusistance is retile. Your ass will be laminated.
kragen 5y ago
> Grey goo. The Borg. Skynet. ... You're reasoning from fictional evidence, like the people who wanted to ban heart and kidney transplants because they'd watched Frankenstein, or who feared computers because they'd watched 2001. Self-reproducing machines are dangerous in the same way that ammonium perchlorate, natural gas, electricity, or computers are dangerous: people can use them to do bad things, and people can engineer systems carelessly, causing accidents. https://en.wikipedia.org/wiki/PEPCON_disaster https://en.wikipedia.org/wiki/PEPCON_disaster https://en.wikipedia.org/wiki/New_London_School_explosion https://en.wikipedia.org/wiki/New_London_School_explosion https://en.wikipedia.org/wiki/War_of_the_currents#Safety_concerns https://en.wikipedia.org/wiki/War_of_the_currents#Safety_con... https://electricenergyonline.com/energy/magazine/362/article/Prevention-of-Oil-Filled-Transformer-Explosions.htm https://electricenergyonline.com/energy/magazine/362/article... https://en.wikipedia.org/wiki/Morris_worm https://en.wikipedia.org/wiki/Morris_worm What is called for is adequate respect for the capabilities and risks of the technology you're working with, prudence in preventing accidents, and avoidance of extreme technological power imbalances like those that led to the Congo Free State and the Google Play Store, not the sort of puerile irrationality that privileges Gene Roddenberry's ideas over those of John von Neumann.
- throwaway316943 5y ago
  Here’s one from reality, bacteria. If we create self replicating machines, even if we do not intentionally introduce variation, they will vary from generation to generation. They will presumably not be indestructible and so some form of selection will be possible. If their reproduction cycle is fast enough they could potentially deviate from their intended purpose in a very short time.
  - Kim_Bruning 5y ago
    In fact, all living things form a category of self replicating machines. Depending, your [highly optimized] self-replicator might have some stiff competition.
    - danielheath 5y ago
unnouinceput 5y ago
Faro plague is the most close to us and the most accurate one from scientific point of view.
imtringued 5y ago
Once you get to the last stage of this paperclip maximizer [0] game your paperclip swarm is slowly being eaten by a competing swarm. Rabbits also reproduce endlessly until predators start hunting them or they starve because of the shortage of food. [0] https://www.decisionproblem.com/paperclips/index2.html https://www.decisionproblem.com/paperclips/index2.html
- TeMPOraL 5y ago
  Yes, but then the universe is nothing but a) paperclips, and b) some residual matter not yet turned into paperclips.

noduerme 5y ago

Not just an off switch, a kill switch. Anything that can reproduce only needs a software modification to change the structure of its hardware on the next generation. If a self reproducing machine's fitness is measured only by real world conditions, and if its software is free to mutate, it will become the dominant microscopic life form very quickly. At that point maybe only an EMP would wipe it out, if it were possible to wipe it out at all. If not, so long carbon based life.

deleted 5y ago
[deleted]
mhb 5y ago
Why not just keep it in sealed hardware that can’t affect the world? https://www.yudkowsky.net/singularity/aibox https://www.yudkowsky.net/singularity/aibox
- someguyorother 5y ago
  There are ways to keep an AI in sealed hardware and making sure it can't affect the world, for instance by using an objective function that only deals with mathematics, and doesn't deal with the real world at all. E.g. the AI is given a fixed amount of hardware and told to produce an algorithm that solves some NP-complete problem (say integer programming) in expected time as close to polytime as possible, as well as a mathematical proof that the algorithm satisfies the claimed close-to-polytime complexity bound. Then humanity can just solve the NP-complete problems separately once they have the algorithm. This objective function doesn't care about the physical world -- it doesn't even know that a physical world exist -- and so it's about as likely to directly affect the physical world as MCTS or AlphaGo. The "AI is going to run out of control" is a very compelling narrative (as everybody who has read the Sorcerer's Apprentice understands). But that doesn't make it true. Beware the availability heuristic. (Incidentally, I think AI destroying mankind because it's too smart is an unlikely outcome. It's much easier for the AI to subvert the human-designed sensors linked to its objective function; and if the AI is sufficiently smart and the sensors aren't perfect, then it can always do so.)
  - mhb 5y ago
    These counterarguments are only possibly effective because you're imagining some particular kind of AI. When there is a useful AI, of course we will want it to be able to interact with people and have it control physical things in the real world. Just like existing computers do.

Iv 5y ago

Making an evolving/learning machine stay in sync with its creators' requirements is called the alignment problem. Publications are made on this. They gave some smarter sounding names to grey goo and paperclip scenarios. An interesting thing to note is that we know how to engineer redundancy and code checking into machines to avoid unwanted mutations. Biology makes us thing this is hard but biological systems do need to evolve, there is no pressure for lower mutations as long as you are at a level that allows a species to survive.

Teever 5y ago
And what if the creators' requirements don't align with the requirements of broader society? We already know that some nation states can behave relatively responsibly with nuclear weapons but we maintain nonproliferation policies with nuclear weapons to prevent rogue nation states and terrorists from wreaking havoc on the world. The destruction that nuclear weapons can cause is nothing compared to the potential destruction from self-replicating machines and it is very unlikely that non-proliferation of self-replicating machines will be impractical if not outright impossible. I am fascinated by youtube channels like Primative Technologies and I am struck by the fact that once self-replicating nanotechnology is developed it is very likely that a properly motivated and well-read individual could walk into a forest with little more than the clothes on their back and walk out with a self-replicating machine, or rather float out with whatever sort of flying machine that their self-replicating nanobots would enable them to build. Once this technology has been invented there's no stopping it. I don't know what we do then. [1] https://www.youtube.com/channel/UCAL3JXZSzSm8AlZyD3nQdBA https://www.youtube.com/channel/UCAL3JXZSzSm8AlZyD3nQdBA
- deleted 5y ago
  [deleted]
- simonh 5y ago
  So first I made a fire drill and start a fire, then dig out some clay and fire it into bricks... *some time later* ... then once the UV laser lithography unit is finished, it's time to assemble the 5nm chip fabrication module for the processing units and Voila. That's the whole thing finished! I jest, but the thing is just because a machine is capable of self replication, that doesn't mean it's any good at making anything other than itself. Organisms are extremely good at making copies of themselves, but they're ultra-optimised for just that. Even the most sophisticated organisms are only able to make a small set of very crude intentional modifications to their environment, if any. We're the only exception. Self replicators and omni-replicators are very different things.

dalf 5y ago

There are biosafety level laboratories, so why not similar labs for self-reproducing machines? But one mistake is not going to go well... Another science fiction example: the Replicators.

Teever 5y ago
With DIY biohacking we're already seeing the trickle down of the kind of work that used to require University or Government level of funding starting to take place in garages and makerspaces. It won't be long before we see the same effect with self-replicating machines.

nyx_land 5y ago

Given the state of the human race right now I think I'd rather give the Borg a fair shot. I'll be the first to volunteer for assimilation.

robbedpeter 5y ago
We are the dyslexic of Gorb. Fusistance is retile. Your ass will be laminated.

kragen 5y ago

> Grey goo. The Borg. Skynet. ... You're reasoning from fictional evidence, like the people who wanted to ban heart and kidney transplants because they'd watched Frankenstein, or who feared computers because they'd watched 2001. Self-reproducing machines are dangerous in the same way that ammonium perchlorate, natural gas, electricity, or computers are dangerous: people can use them to do bad things, and people can engineer systems carelessly, causing accidents. https://en.wikipedia.org/wiki/PEPCON_disaster https://en.wikipedia.org/wiki/PEPCON_disaster https://en.wikipedia.org/wiki/New_London_School_explosion https://en.wikipedia.org/wiki/New_London_School_explosion https://en.wikipedia.org/wiki/War_of_the_currents#Safety_concerns https://en.wikipedia.org/wiki/War_of_the_currents#Safety_con... https://electricenergyonline.com/energy/magazine/362/article/Prevention-of-Oil-Filled-Transformer-Explosions.htm https://electricenergyonline.com/energy/magazine/362/article... https://en.wikipedia.org/wiki/Morris_worm https://en.wikipedia.org/wiki/Morris_worm What is called for is adequate respect for the capabilities and risks of the technology you're working with, prudence in preventing accidents, and avoidance of extreme technological power imbalances like those that led to the Congo Free State and the Google Play Store, not the sort of puerile irrationality that privileges Gene Roddenberry's ideas over those of John von Neumann.

throwaway316943 5y ago
Here’s one from reality, bacteria. If we create self replicating machines, even if we do not intentionally introduce variation, they will vary from generation to generation. They will presumably not be indestructible and so some form of selection will be possible. If their reproduction cycle is fast enough they could potentially deviate from their intended purpose in a very short time.
- Kim_Bruning 5y ago
  In fact, all living things form a category of self replicating machines. Depending, your [highly optimized] self-replicator might have some stiff competition.
  - danielheath 5y ago

unnouinceput 5y ago

Faro plague is the most close to us and the most accurate one from scientific point of view.

imtringued 5y ago

Once you get to the last stage of this paperclip maximizer [0] game your paperclip swarm is slowly being eaten by a competing swarm. Rabbits also reproduce endlessly until predators start hunting them or they starve because of the shortage of food. [0] https://www.decisionproblem.com/paperclips/index2.html https://www.decisionproblem.com/paperclips/index2.html

TeMPOraL 5y ago
Yes, but then the universe is nothing but a) paperclips, and b) some residual matter not yet turned into paperclips.