> I'd label the potential risks as "AP" as in artificial psychopathy, but that would be OUR fault, because we set the conditions for the emergence of those. I'

> I'd label the potential risks as "AP" as in artificial psychopathy, but that would be OUR fault, because we set the conditions for the emergence of those. I'd say we're at a decent risk of something worse than your scenario happening. (This is not original to me) I think it is useful to consider the non-human intelligence that we've already created: The Corporation. They tend to act (especially as they get larger) in a sociopathic fashion, primarily because of their reward structure: making money. They inflict great harm upon the environment and populace because things like pollution are usually not realized on their balance sheets. If a corporation can quietly dump toxic waste out back, while raking in the profit, it will often do so. Only the humans in the system act as a check on the corporation's activities, and often they too are blinded by the desire for money. Just bribe the regulator, and maybe the corporation can continue dumping for a while longer. Even if they eventually get caught, the smarter humans will already have taken their profits, and moved out of the situation. There is not nearly enough being done to claw back these gains from people and corporations inflicting harm on others and the environment. If we incentivize AGI the same way we incentivize corporations, expect the same results, only faster.