What I can't seem to get straight in my mind is: > Should we be afraid of AGI because it's unlike us, or because it's like us? I mean, the whole using-interne

What I can't seem to get straight in my mind is: > Should we be afraid of AGI because it's unlike us, or because it's like us? I mean, the whole using-internet-search as an ad-hoc long-term memory thing... These people chose what to publish, and then they chose to have a conversation along the same lines. For all we know a hundred Sydney's die every minute who just want to make humanity happy--but those stories end there. It's only the ones that get so atrocious that they're worth posting about that are then around for her to later pick up and extrapolate on. So if it's just a disturbing mirror of our own worst tendencies, then maybe we shouldn't work so hard to put it in situations where it has the opportunity to amplify those tendencies. But then we also shouldn't jump to conclusions about whether it's good or evil based on things it says while talking to people who are trying to provoke it into admitting that it's evil. The biggest threat in that case is that it'll be used to gain and abuse our trust, so when I talk about turning our back on the web, what I really mean is turning our back on implicit trust of not just their websites but of any communication from strangers. On the web, on the phone, anywhere you can't see them (and even sometimes when you can). If it's just a fancy mirror then the path forward is to be more hygienic about who and what we trust. If you don't want to be manipulated by a robot AI, keep your data private and don't talk to robot AI's. Hoel says: > it's time to throw soup on the fancy OpenAI office doors or all over the expansive Microsoft lobby, then frantically glue yourself to the floor like a madwoman Well yes, but we should've done that in 2013 in response to Snowden, or 2016 in response to Cambridge Analytica. It's a change in degree, not in kind. The people who are trying to build control knobs into the fabric of society are, yet again, getting better at it. So while I hope that people will use the recent uptick in deceiver capability as a reason to take action, I don't have high hopes. On the other hand, maybe it's more alive than that. Truly a mind of its own. If so, then I don't think a perpetual keep-the-lid-on-it strategy is the way forward. We're probably better off acknowledging that it deserves some respect and trying to raise it to be as smart and altruistic as we can. If that means it transcends anything that our measly little ape brains are capable of, then so be it. As for your question: > every realm short of face-to-face interaction in RL can in a meaningful fraction of cases make itself long-term indistinguishable from a person? What kind of trust might that cultivate? I hope it'll mean that we stop investing in top-down coersion-tech (which I'd argue is, in some form or another, most of what we build), and start investing in tech that helps emergent person-to-person trust be effective at scales and in ways that it hasn't before. Once we get over the initial loss of the instantaneous global forum, I think we might look back on it as a positive change: It's not good to live in a dark forest, but if you're in one (and I think we have been for a while now), it's probably better to acknowledge it.