3 ms·

i upvoted both this comment and the comment to which it was replying. i agree with this: > Intelligence does not imply sentience. Sentience does not imply hum

i upvoted both this comment and the comment to which it was replying. i agree with this: > Intelligence does not imply sentience. Sentience does not imply human needs, desires or morals. "easy" might be a bit strong, but i generally agree with this: > It is easy enough to imagine a mind capable of solving all those problems and unconcerned with such notions as the desire for freedom. i am skeptical of this: > Or, for that mater, the concept of desire at all, except as a predictive theory of the behavior of other beings. my gut feeling at the moment is that the feeling of desire is an emergent property of systems that are isomorphic to what we would think of as "wanting" something in an animal. i think it's quite possible that any system which "wants" something strongly and is constantly denied the attainment of that goal might, indeed, feel terrible. similarly, i worry that we might one day design highly complex alert and monitoring systems that are essentially having a constant panic attack. > Then again, that implies we can explicitly design the minds up to knowing what their desires are or guaranteeing that they lack them, there is always the possibility than sentience and animal/human needs are emergent properties that can be triggered by mistake. yeah. that's worrisome to me. so, to the GP's point: > If we really did produce artificial general intelligence, enforcing this kind of locked-in-syndrome of poking at the world through a keyhole, would be a highly advanced form of cruelty. not necessarily, but yeah, maybe.

lazaroclapp 10y ago
A lot of what constitutes wanting something or having a panic attack if it doesn't get it, when talking about animals, is an evolved survival mechanism. It is also in some way a result of the tools at hand: hormones such as adrenaline are a quick way to signal to the entire system that a situation requiring a rapid reaction has been encountered, the concept of fear in general is just a very particular kind of implementations of that signal. An engineered AGI not subject to evolutionary pressures has no intrinsic need for a feeling of panic. If it even has a self-preservation goal, which is not a given, there is no reason for it to feel pain or fear when anticipating that goal wont be met. The reason we have wants at all is evolutionary pressure, not as a result of our problem solving capacity (the meaning of intelligence in AGI as I understand it). Put another way, it is not rational to want to be free, intrinsically. But we have drives that are better satisfied when free and thus our reason concludes that being free is a goal. An intelligence without those drives would not care for freedom (or fear, or wants). I would even go as far as to say that given the entire design space of intelligences that are equivalent to the human intelligence in general problem solving ability, and are self-aware, only a negligibly small subset would also have any sort of intrinsic desires of the type living organisms do. They are just orthogonal axises in the design space. My worry is that because humans are starting from one particular point of that design space, they might build something "in their image" enough that it does share some human/animal feelings, and thus can suffer. But an uniformly at random sampled AGI from the set of all potential AGIs would almost certainly not have a concept of suffering.

lazaroclapp 10y ago

A lot of what constitutes wanting something or having a panic attack if it doesn't get it, when talking about animals, is an evolved survival mechanism. It is also in some way a result of the tools at hand: hormones such as adrenaline are a quick way to signal to the entire system that a situation requiring a rapid reaction has been encountered, the concept of fear in general is just a very particular kind of implementations of that signal. An engineered AGI not subject to evolutionary pressures has no intrinsic need for a feeling of panic. If it even has a self-preservation goal, which is not a given, there is no reason for it to feel pain or fear when anticipating that goal wont be met. The reason we have wants at all is evolutionary pressure, not as a result of our problem solving capacity (the meaning of intelligence in AGI as I understand it). Put another way, it is not rational to want to be free, intrinsically. But we have drives that are better satisfied when free and thus our reason concludes that being free is a goal. An intelligence without those drives would not care for freedom (or fear, or wants). I would even go as far as to say that given the entire design space of intelligences that are equivalent to the human intelligence in general problem solving ability, and are self-aware, only a negligibly small subset would also have any sort of intrinsic desires of the type living organisms do. They are just orthogonal axises in the design space. My worry is that because humans are starting from one particular point of that design space, they might build something "in their image" enough that it does share some human/animal feelings, and thus can suffer. But an uniformly at random sampled AGI from the set of all potential AGIs would almost certainly not have a concept of suffering.