3 ms·
If a hacker designs an AI which has among its talents "self-replicating to another host", did an AI escape containment, or did a hacker write a virus? The alig
by jazzyjackson 4y ago
If a hacker designs an AI which has among its talents "self-replicating to another host", did an AI escape containment, or did a hacker write a virus?
The alignment concern falls flat with me because it assigns agency to a computer program. At what point is an AI's intentions its own, and not a cost function put in place by a programmer, directed by an investor? I am more concerned about the actions of programmers and investors than some theoretical virtual self.
- dotsam 4y agoI definitely don't want to sound like I'm saying we should stop worrying about hackers and bad actors, they are of course a concern. I certainly don't want to advocate for dismissing real dangers and power structures in the real world in favour of speculation over some theoretical risk. But I do want people to think about it and take it seriously, in the same way I would have wanted people in 1933 to take seriously the possibility of nuclear weapons even though they were then only theoretical. I think the sticking point in this is that you don't think that AGI is ever possible - is that right? If so, I know a comment on a HN thread is very unlikely to change your mind on it! But when the stakes of being wrong are high, I find it useful to go from a position of "it won't happen, so I won't worry about it" to something like "on balance I'm pretty sure it won't happen, but I may be wrong, and in that case I would be worried about it". I'd then feel much better about spending some time researching the topic in more depth.