We can't agree on a non-leaky definition of benefiting humanity, and we can't be sure of a lossless way of communicating this ideal to an AGI if we could agree.

We can't agree on a non-leaky definition of benefiting humanity, and we can't be sure of a lossless way of communicating this ideal to an AGI if we could agree. We then don't know how our instructions will be interpreted (is it just pretending to understand), or if it will understand the concept of "greater good", which would allow it to justify to itself doing practically anything towards any stated goal. On top of this, the people trying to race to this eventuality are trying to convince us that any of this is possible to constrain. Despite binging enough Ghost in the Shell to entertain all of the scenarios I still don't think AGI is anywhere near and I'm just going to mostly ignore the possibility. One of those distant existential things I have no control over so I'll try not to think about it too much :)