I think the key point is that our species has been shaped by genetic adaptations to live in stable groups of humans. We have innate empathy; we automatically po

I think the key point is that our species has been shaped by genetic adaptations to live in stable groups of humans. We have innate empathy; we automatically ponder about correction signals from other humans and correct our behaviors accordingly; and collaboration simply makes sense to us because it helps to protect and improve our own fragile lives. The scary prediction about AI is that it might be feasible, if not easy, to build an AI that is simply programmed to maximize certain reward signals (e.g. to clean the floor) without respecting limitations that lead to a stable coexistence with humans and other AIs. I think a main problem might be that a super-intelligence would not necessarily need to care about collaboration. If the SI can figure out ways to simply synthesize everything it needs from existing matter, then collaboration with us would not make sense. The floor cleaning AI might figure out ways to remove absolutely everything from all floors on planet Earth (including humans), because that is perfectly consistent with its goals and it also maximizes its reward signals. That would certainly not make our planet prettier than we are able to shape it. This is comparable, but not the same as corruption in powerful organizations such as BP. They can make their own rules about cleaning up the Deepwater Horizon oil spill because there are few correction signals coming from above. If you are at the top, you can play by your own rules. In case of human organizations, there are still limitations though. There is still a need for collaboration, for example dependence on the medical industry in case the CEO gets sick. Also BP cannot afford to scare away all they customers and workers, at least until BP is fully automated and can synthesize everything it needs.