I think there's some over-anthropomorphization of the AI here, in terms of it picking up human qualities as if they were Things outside our brains that we posse

I think there's some over-anthropomorphization of the AI here, in terms of it picking up human qualities as if they were Things outside our brains that we possess. But I feel like this particular argument is (please correct me if I'm wrong) saying something like "Well of course we wouldn't let them go crazy! That'd be ridiculous!" I agree -- a sensible solution would be to have good design and introduce safeguards. That's the point of people like Bostrom when they're advocating for this, because before they started writing about this, it seems like a lot of people weren't even thinking it could be a problem. How many systems will be built without good design, because they were made to solve a short-term problem. Will the finance bots doing transactions at the microsecond scale in order to maximize the assets of some particular company have been built to not let them go haywire?