The biggest problem with possible future AGI/ASI are not the possibilities, but that all the feedback loops are closed, meaning what we think about it, and what

The biggest problem with possible future AGI/ASI are not the possibilities, but that all the feedback loops are closed, meaning what we think about it, and what computers think about it, changes the outcome. This sets up a classic chaotic system, one extraordinarily sensitive to initial conditions. But it's worse. A classic chaotic system exhibits extreme sensitivity to initial conditions, but this system remains sensitive to, and responds to, tiny incremental changes, none predictable in advance. We're in a unique historical situation. AGI boosters and critics are equally likely to be right, but because of the chaotic topic, we have no chance to make useful long-term predictions. And humans aren't rational. During the Manhattan Project, theorists realized the "Gadget" might ignite the atmosphere and destroy the planet. At the time, with the prevailing state of knowledge, this catastrophe had been assigned a non-zero probability. But after weighing the possibilities, those in change said, "Hey -- let's set it off and see what happens."