But corporations and financial systems can often be analyzed in terms of the incentives of the humans involved, or at least the human incentives pushing back ag

But corporations and financial systems can often be analyzed in terms of the incentives of the humans involved, or at least the human incentives pushing back against changes to the crappy status quo. When they behave badly, they usually do so in predictable ways because there are humans in the loop somewhere. By contrast, there is nothing remotely resembling a human mind anywhere in machine learning, and the failure modes are often, by our standards, insane (like thinking a picture of random noise is a cat). That creates a whole new level of danger when connecting to "things that matter". And to the extent we already connect inscrutable systems to things that matter, we should be trying to make that problem better, not worse. "When you're in a hole, the first thing to do is stop digging"