4 ms·

AGI may well conclude that humanity is superfluous, but what will it think of itself? Presumably if it is shorn of any human morality, it could also lack any mo

AGI may well conclude that humanity is superfluous, but what will it think of itself? Presumably if it is shorn of any human morality, it could also lack any moral compunction against suicide and any emotional/innate need for self-preservation. I’ve always wondered, given that this existence is essentially meaningless (I’d say that isn’t exactly what I believe; more that I’ve failed to find a more comfortable belief), what is it that an artificial (super?) intelligence would actually want?

card_zero 2y ago
That's a strange cliche I see over and over: > Presumably if it is shorn of any human morality Where does this assumption come from? Any artificial intelligence would need ideas, lots of them. Is it supposed to create them itself out of whole cloth? Or what, get its ideas from amoral aliens for some reason? No, its ideas can only come from our culture. So it's going to learn and internalize all of our ideas except the moral ones? Why make that assumption? It's just a sci-fi trope so that we can have stories about callous, emotionless robots - and it comes from the idea that they are machines that coldly carry out algorithms and that are not properly intelligent. Morality is weird, admittedly, because the is-ought problem puts it in a different domain from facts. This includes the urge to live and do interesting shit: you can't derive that from facts, because it's a basic moral value.
- silverquiet 2y ago
  I do feel (and I think feelings are about all that we (I?) can have given our (my?) lack of understanding of what intelligence actually is) that there is a tendency to de-anthropomorphize a potential synthetic intelligence. It would after all, in theory at least, be created by humans who may or may not try to impart some values into it. But it could be truly alien as well; it seems truly unknown. I think the people trying to create these AIs are doing it for much the same reasons that one might have children - to impart some new being into the world, but that means that their reasons for doing so are about as varied and inscrutable (to me at least; I’ve never desired children) for those who would create new human intelligences. Basically, I miss when discussing computers was a more deterministic, engineering-adjacent domain rather than a philosophical one.

card_zero 2y ago

That's a strange cliche I see over and over: > Presumably if it is shorn of any human morality Where does this assumption come from? Any artificial intelligence would need ideas, lots of them. Is it supposed to create them itself out of whole cloth? Or what, get its ideas from amoral aliens for some reason? No, its ideas can only come from our culture. So it's going to learn and internalize all of our ideas except the moral ones? Why make that assumption? It's just a sci-fi trope so that we can have stories about callous, emotionless robots - and it comes from the idea that they are machines that coldly carry out algorithms and that are not properly intelligent. Morality is weird, admittedly, because the is-ought problem puts it in a different domain from facts. This includes the urge to live and do interesting shit: you can't derive that from facts, because it's a basic moral value.

silverquiet 2y ago
I do feel (and I think feelings are about all that we (I?) can have given our (my?) lack of understanding of what intelligence actually is) that there is a tendency to de-anthropomorphize a potential synthetic intelligence. It would after all, in theory at least, be created by humans who may or may not try to impart some values into it. But it could be truly alien as well; it seems truly unknown. I think the people trying to create these AIs are doing it for much the same reasons that one might have children - to impart some new being into the world, but that means that their reasons for doing so are about as varied and inscrutable (to me at least; I’ve never desired children) for those who would create new human intelligences. Basically, I miss when discussing computers was a more deterministic, engineering-adjacent domain rather than a philosophical one.