Think about modern financial trading algorithms -- they're probably built to maximize the money that flows in, and there's nothing that says "I know that this m

Think about modern financial trading algorithms -- they're probably built to maximize the money that flows in, and there's nothing that says "I know that this money will be used for X." I think financial algorithms are probably the best non-toy model of dangerous AI, simply because it's easy to see the flow from optimization to real-world consequences. The problems with end-of-the-world AGI scenarios are practical, not theoretical, and I think a lot of people get hung up on arguments such as, "Of course we wouldn't allow a paperclip-manufacturing AI the ability to deconstruct human beings." In a larger sense, I think Bostrom can restated (in part!) as the trivial observation that "optimization is insensitive to hidden variables," and it describes any kind of complex system, not AI and AGI alone. For example, the failure to capture environmental remediation costs in market pricing structures means that we improperly discount future costs for things such as global warming or lead exposure. In health care, the implementation of the Prospective Payment System led to hospitals dumping patients back on the street "quicker and sicker" to maximize margins on care. And thousands of financial systems focused on microsecond returns don't pay attention to, or even have any way of understanding, how their coordinated activities can be macroeconomically destabilizing. Any time you allow outcome optimization+general reasoning+real-world outputs, you have the opportunity for a killer paperclip scenario.