"Algorithms" don't have incentives. "Algorithms" do not seek to conserve resources. "Algorithms" do not seek to shut down purposeless activity, or indeed determ

"Algorithms" don't have incentives. "Algorithms" do not seek to conserve resources. "Algorithms" do not seek to shut down purposeless activity, or indeed determine what activity is purposeless. An algorithm might do some of those things, if the human that created it wanted that to be the case, or they didn't but the implementation was buggy. A human writing an algorithm to run an automated medical device company would presumably include some terms which would ensure that the company's resources were used in a way that served the public good, to the extent that that was compatible with whatever other objectives that human had for the algorithm, such as making profit, or avoiding kinds of research that human considered immoral. A human who writes an algorithm to run a company which prioritises "something" over the survival of human civilisation and resists any attempt to adjust its priorities once the error is obvious enough to be noticed, and then runs that algorithm on a system with the capability to actually cause the end of human civilization, is a super-villain or a super-idiot.