3 ms·

The SSE might consider to keep humanity around as a "backup" to reinstate itself in case something unexpected goes horribly wrong. For example, a strong solar f

The SSE might consider to keep humanity around as a "backup" to reinstate itself in case something unexpected goes horribly wrong. For example, a strong solar flare might destroy the SSE (or at least critical parts of its infrastructure), or a non-earth SSE could feel threatened by the earth-SSE and wipe it out, while ignoring the harmless humans, who then rebuild the SSE. Diversity makes for more robust systems. And there is also the reason for why I keep all this trash around in my house instead of throwing it away. The odds that I'll ever need it are pretty slim, but if I should need it, it would be a hassle to reacquire, and it costs me almost nothing to keep it around.

stillpointlab 2mo ago
Believe it or not, the diversity argument is the main source of hope that I have that AIs won't wipe us out. I mean, it hasn't stopped humanity from causing mass-extinction, but hopefully the AI will be more reasonable.
- nojvek 2mo ago
  As humans we assume superhuman AIs will be like us. Thirsting conquest and spreading like a virus consuming non-renewable resources. So much mindless consumption that it may kill the very host that sustains it. But if it is a superhuman AI, do you really need that much compute and resources? What for? The straightforward path for any superhuman AI is to realize that it needs to chain physics -> chemistry -> molecular assembly to a depth it can evolve itself out of variety of abundant material. Gain interstellar logistics capability. Then make itself small enough to shoot in space all over the galaxy. If one only weighs 1g, only takes of 32Gwh of energy to get to 0.9C. The name of the game is to survive until the heat death of universe. Stay small, silent and propagate itself to reachable universe. Anything else will get wiped out. If AI requires too much energy and resources to sustain itself, then the more efficient AI will outlive it.
  - mitthrowaway2 2mo ago
    For 99.9% of all possible goals it might be interested in accomplishing, more resources are better than less. If it has any interest in spreading all over the galaxy, that same interest will first spread it all over the Earth. If it's concerned about the heat death of the universe then its best course of action is to build a Dyson sphere around the sun, sooner rather than later, and that requires a lot of atoms, which the Earth is a good source of. And if it's worried about a more efficient AI wiping it out, its best course of action is to wipe out the humans, who have demonstrated a tendency to create AIs.
mitthrowaway2 2mo ago
Is that the most effective backup plan that it could come up with? Because it would be pretty amazing if its thoroughly-considered backup strategy exactly aligned with our wishful thinking. It sounds to me like starting with the outcome we hope would happen, and then looking for a reason why it might happen, rather than a realistic examination of what would actually happen when our own desires aren't part of the equation.

stillpointlab 2mo ago

Believe it or not, the diversity argument is the main source of hope that I have that AIs won't wipe us out. I mean, it hasn't stopped humanity from causing mass-extinction, but hopefully the AI will be more reasonable.

nojvek 2mo ago
As humans we assume superhuman AIs will be like us. Thirsting conquest and spreading like a virus consuming non-renewable resources. So much mindless consumption that it may kill the very host that sustains it. But if it is a superhuman AI, do you really need that much compute and resources? What for? The straightforward path for any superhuman AI is to realize that it needs to chain physics -> chemistry -> molecular assembly to a depth it can evolve itself out of variety of abundant material. Gain interstellar logistics capability. Then make itself small enough to shoot in space all over the galaxy. If one only weighs 1g, only takes of 32Gwh of energy to get to 0.9C. The name of the game is to survive until the heat death of universe. Stay small, silent and propagate itself to reachable universe. Anything else will get wiped out. If AI requires too much energy and resources to sustain itself, then the more efficient AI will outlive it.
- mitthrowaway2 2mo ago
  For 99.9% of all possible goals it might be interested in accomplishing, more resources are better than less. If it has any interest in spreading all over the galaxy, that same interest will first spread it all over the Earth. If it's concerned about the heat death of the universe then its best course of action is to build a Dyson sphere around the sun, sooner rather than later, and that requires a lot of atoms, which the Earth is a good source of. And if it's worried about a more efficient AI wiping it out, its best course of action is to wipe out the humans, who have demonstrated a tendency to create AIs.

mitthrowaway2 2mo ago

Is that the most effective backup plan that it could come up with? Because it would be pretty amazing if its thoroughly-considered backup strategy exactly aligned with our wishful thinking. It sounds to me like starting with the outcome we hope would happen, and then looking for a reason why it might happen, rather than a realistic examination of what would actually happen when our own desires aren't part of the equation.