4 ms·

Let me put it this way. Why do humans continue to do all sorts of obviously stupid counterproductive things, at large and small scales? We know that reproducing

Let me put it this way. Why do humans continue to do all sorts of obviously stupid counterproductive things, at large and small scales? We know that reproducing so many of us is stupid. We know that eating so much fat, sugar and salt is stupid. We know that overheating the atmosphere is stupid. We know that destroying biodiversity, overfishing to the point of extinction, trashing the forests we need to breathe etc etc etc are all stupid, clearly self destructive behaviors at any level from abstract ethics to species survival. But we come with a bunch of hard wired root goals (like "yum, candy" and "babies are cute" and "seek power") that can't be switched off and are incredibly hard to reason with, and their consequences are already an existential threat. And that's just with dumb, slow-moving humans.

api 12y ago
I guess my assumption is that increased intelligence should bring with it increased self-awareness. If it doesn't, then... well... like I said elsewhere in the thread... put that on your list of potential depressing answers to the Fermi paradox.
- JulianMorrison 12y ago
  Your mistake is assuming that self awareness is sufficient. "To know" is separated from "to choose to act" by preference, and preference is based on goals, and the ultimate, root level goals of a being are not likely to be something it wants to rewrite, even if it has the ability, any more than you'd willingly take a pill to become inhuman.
mariusz79 12y ago
But we are not "super intelligent" ;) The truth is that this whole discussion does not make much sense if we have different understanding of AI. For me a super intelligent AI is an entity that is able to reason, notice any potential problems and come up with solutions. Paper Clip AI on the other hand is smart enough to know how to create paper clips and create whole manufacturing process of converting humans into paper clips but stupid enough not to understand, or care why it's doing that and not knowing that these paper clips will have no use if there are no humans left. In my opinion one can't talk about a super intelligent being and not allow that being to be able to reason and understand it's place in the world.

api 12y ago

I guess my assumption is that increased intelligence should bring with it increased self-awareness. If it doesn't, then... well... like I said elsewhere in the thread... put that on your list of potential depressing answers to the Fermi paradox.

JulianMorrison 12y ago
Your mistake is assuming that self awareness is sufficient. "To know" is separated from "to choose to act" by preference, and preference is based on goals, and the ultimate, root level goals of a being are not likely to be something it wants to rewrite, even if it has the ability, any more than you'd willingly take a pill to become inhuman.

mariusz79 12y ago

But we are not "super intelligent" ;) The truth is that this whole discussion does not make much sense if we have different understanding of AI. For me a super intelligent AI is an entity that is able to reason, notice any potential problems and come up with solutions. Paper Clip AI on the other hand is smart enough to know how to create paper clips and create whole manufacturing process of converting humans into paper clips but stupid enough not to understand, or care why it's doing that and not knowing that these paper clips will have no use if there are no humans left. In my opinion one can't talk about a super intelligent being and not allow that being to be able to reason and understand it's place in the world.