Consider occupational safety. A table saw isn't self aware and just a powerful tool that cuts whatever you put in the path of its blade. If someone puts their f

Consider occupational safety. A table saw isn't self aware and just a powerful tool that cuts whatever you put in the path of its blade. If someone puts their finger there and the saw cuts it off, the problem with safety is the guy with the finger, not the saw, right? But in reality, people know that they might end up being the guy with the finger, and they would like to keep that finger, so they use a saw with an automatic stop mechanism that saves the finger at the cost of destroying the blade. Wanting your tools to not hurt you isn't so strange, is it? Of course current AIs couldn't chop off your finger even if they tried, let alone build a nuclear bomb, but that doesn't mean wanting to keep it that way is an inherently stupid proposition.