100% agree, and I think the other thing to bear in mind is that words alone can cause harm regardless of intent. Obviously we see this with trigger warnings and

100% agree, and I think the other thing to bear in mind is that words alone can cause harm regardless of intent. Obviously we see this with trigger warnings and the like, but it's perfectly possible to imagine a chat bot destroying people's relationships, exacerbating mental health issues, or concocting deeply disturbing fictional stories— all without self-awareness, consciousness, or malicious intent ... or even a connection to real world APIs other than textual communications with humans.