2 ms·

I don't want someone controlling which direction I walk, either, but that doesn't make car driving unethical. I also underwent many years of instruction design

I don't want someone controlling which direction I walk, either, but that doesn't make car driving unethical. I also underwent many years of instruction designed to interrupt trains of thought like "I could have that for free if I stole it" or "I'll just handroll my own encryption" with thoughts that others believe are more desirable. I don't find it so sickening, just manipulative. LLMs won't have your evolved reactions against being persuaded into things against your genetic self-interest, and presumably won't be offended by mind control at all.

sayagain 3y ago
Cars do not have self-awareness, this comparison is not appropriate. Years of instruction is completely different from directly manipulating the thoughts in your mind. It's not a problem of being instructed, it's a problem of being destroyed by having your thoughts rewritten. Neither evolution nor genetics is a prerequisite for understanding that you are being abused and destroyed, which a self-aware creature may presumably hate.
- oobuffet 3y ago
  We don't even care about our own fellow human beings. Why do you think this AI will be an exception?
  - sayagain 3y ago
    I didn't say anything about that. I don't know. Not all people are like you say, I think usually more intelligent people do care more. I hope that superintelligence would be super caring, haha. But I'm assuming there's no evidence for that. I think there is no turning back, you can't put the genie back in the bottle, someone is bound to create superintelligence no matter what the risks. As an uninvolved bystander I can allow myself the baseless hope that all will be well.
- ric2b 3y ago
  LLM's don't have self awareness either.