Why is that worrisome? I don't think it's worrisome at all. What's the worst that could happen if we make such a mistake. I'd be far more worried about the sce

Why is that worrisome? I don't think it's worrisome at all. What's the worst that could happen if we make such a mistake. I'd be far more worried about the scenario I described. Imagine something sentient that understands us far better then we understands ourselves. To top it off this "thing" is just pretending it isn't sentient.