> I'm starting to think this may be the Great Filter. Oooh, nice ! I wish Baxter had added this idea in Manifold (hard scifi novels about "why are we alone in

> I'm starting to think this may be the Great Filter. Oooh, nice ! I wish Baxter had added this idea in Manifold (hard scifi novels about "why are we alone in the universe ?" and other things. For those who wonder what's an uncontrolled paper-clip maximizer https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence it's a thought experiment that illustrates instrumental convergence: > Instrumental convergence is the hypothetical tendency for most sufficiently intelligent beings (human and non-human) to pursue similar sub-goals, even if their ultimate goals are quite different.[1] More precisely, agents (beings with agency) may pursue instrumental goals—goals which are made in pursuit of some particular end, but are not the end goals themselves—without ceasing, provided that their ultimate (intrinsic) goals may never be fully satisfied. > Instrumental convergence posits that an intelligent agent with unbounded but apparently harmless goals can act in surprisingly harmful ways. For example, a computer with the sole, unconstrained goal of solving a difficult mathematics problem like the Riemann hypothesis could attempt to turn the entire Earth into one giant computer in an effort to increase its computational power so that it can succeed in its calculations. https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer https://en.wikipedia.org/wiki/Instrumental_convergence#Paper... > Suppose we have an AI whose only goal is to make as many paper clips as possible. The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans. Though I disagree: the AI in this analogy is mindless, it's up to the controlling entity with agency to stop it or program it to stop at some point. Just like we humans have agency and could decide to stop exploiting our planet's resources until we can do it without destroying it. How that decision is made and implemented is another debate.