The fears around AGI seems to be based on the thinking that AGI will be like humans, but smarter. We kill each other for resources, so AGI will kill us (perhaps

The fears around AGI seems to be based on the thinking that AGI will be like humans, but smarter. We kill each other for resources, so AGI will kill us (perhaps inedvertantly) to secure resources for itself. This ignores that intelligence will necessarily be moulded by the environment in which it evolves, and AGI will essentially be created through a process of evolution. Maybe AI will design AGI, but the results will be carefully selected, by humans, based on AGI's usefulness to us. So while we were shaped by evolution to compete against each other for resources, AGI will be shaped by evolution to compete against each other to please humans (to gain what it considers resource: computation time). The dangers of AGI can perhaps be thought of in terms of 1984 vs Brave New World.. AGI might be dangerous because of how effective it will be at satisfying us.. perhaps making us somhow completely addicted to whatever the AGI produces. This process will probably also lead to AGI and humanity merging. The most effient way to serve a human is to interface with it more efficiently. AGI might used to help design better neural interfaces. I think we may underestimate how much "randomness" is involved in seemingly intelligent decisions by humans. We do a lot of stupid shit, but sometimes we get lucky and what we did looks really smart. I suspect the environment in which we develop AIs (unless it's in games) will be too rigid. A paperclip maximizer might consider if it should work with humanity to produce the most paperclips, or just violently take over the whole planet. But that may be undecidable. It could never get enough information to gain confidence in what's the better decision. We humans have those handy emotions to cut through the uncertainty, and decide that it's better to kill those other humans and grab their stuff because they're not in our social groups, they're bad people.