> "AI risk is a concern because less intelligent entities are not usually able to constrain the behavior of more intelligent entities" - A coarse and imperfect

> "AI risk is a concern because less intelligent entities are not usually able to constrain the behavior of more intelligent entities" - A coarse and imperfect model indeed, but certainly a model based on observations of interactions in the real world. I'd argue that the observations are inconclusive. Human dominion over nature is a holistic thing that involves more than just our intelligence -- the particular shape and capabilities of the human body (opposable thumbs, upright posture, stamina) certainly are an important enabling factor without which our intelligence would probably have been pissing in the wind. Within human societies, I see little evidence that more intelligent humans can effectively constrain the behavior of less intelligent humans. Who can manipulate who seems to be more correlated with personality and instinct than intelligence per se, and it's quite failure prone. There are also classes of powerful non-intelligent systems such as viruses and grey goo that intelligence could foil, but not necessarily: it is perfectly possible, I dare say plausible, that there exists a class of non-intelligent systems that no physically realizable intelligence can control. I feel that we overestimate the role of intelligence in our success, as well as its scope (more precisely, I think we ignore or underplay the specific circumstances that make it effective). I mean, intelligence isn't magic, it's a system that attempts to model the world and plan how to get from the current world state to some desired state. It stands to reason that intelligence isn't going to scale well with degrees of freedom, regardless of how it is implemented, because the size of the search space explodes as complexity increases. In order to work properly, intelligence needs a world simple enough to model reliably. If it isn't simple enough, simplify it by force. If that can't be done, tough luck. I would argue that human society, as a system, actually is intractable for human intelligence, which is why smart people are so bad at understanding it, let alone steering it. Now, would it be tractable for AI? I have my doubts, but what worries me here isn't the level of intelligence. After all, we don't dominate nature because we understand it very well. We dominate it through brute force. We have the minimum level of intelligence to create systems that can overwhelm natural processes, which turned out to require comparatively very, very little complexity. That's kind of lucky. I don't know if AI would have that luck -- in any case, it is best to proceed cautiously.