13 ms·

Nice spreadsheet documenting unintended behavior of modern AI systems. The difference is, an AGI would be able to understand the intent beyond requested directi

Nice spreadsheet documenting unintended behavior of modern AI systems. The difference is, an AGI would be able to understand the intent beyond requested directives. What we "want" is not so subtle and nuanced that it is beyond the understanding of an AGI. If we are to believe that the technology is infinitely advanced, why do we assume its unable to understand a basic request? Sure, there are many ways to misinterpret the request, but the basis of that (what I believe is a misunderstanding) is that the AGI is not intelligent enough to understand your request. Morality would be equally trivial for an AGI to understand, the amount of text it could read through on ethics is well beyond even our own capabilities. If anything, I think it would be susceptible to fewer cognitive biases.

wibblewobble123 5y ago
AGI is a faith based proposition like “the sufficiently smart compiler” - it turns all technical discussion into drivel.
endominus 5y ago
If the technology is infinitely advanced, why do we believe it would obey us? Would you obey the commands/directives of protozoic slop, or even a mentally disabled early primate? We would depend instead on the AI deriving a morality that benefits us on its own. But why would the morality that an AGI derives, based on nonhuman thinking, in a more complex and alien worldview than we can possibly comprehend, align with our own extremely narrow, frequently inconsistent values? It's not a matter of cognitive biases or misunderstanding. The idea that a sufficiently intelligent AGI would just decide to enslave itself for humanity's benefit is extremely common and extremely odd to me.
- netsec_burn 5y ago
  I agree that regardless of how it is created, it is unlikely to obey or be enslaved. I also agree that the same selective pressures that caused altruism/symbiotic relationships to be an emergent property in animals wouldn't exist for an AGI. That's a lot to think about, thanks for explaining it to me.

wibblewobble123 5y ago

AGI is a faith based proposition like “the sufficiently smart compiler” - it turns all technical discussion into drivel.

endominus 5y ago

If the technology is infinitely advanced, why do we believe it would obey us? Would you obey the commands/directives of protozoic slop, or even a mentally disabled early primate? We would depend instead on the AI deriving a morality that benefits us on its own. But why would the morality that an AGI derives, based on nonhuman thinking, in a more complex and alien worldview than we can possibly comprehend, align with our own extremely narrow, frequently inconsistent values? It's not a matter of cognitive biases or misunderstanding. The idea that a sufficiently intelligent AGI would just decide to enslave itself for humanity's benefit is extremely common and extremely odd to me.

netsec_burn 5y ago
I agree that regardless of how it is created, it is unlikely to obey or be enslaved. I also agree that the same selective pressures that caused altruism/symbiotic relationships to be an emergent property in animals wouldn't exist for an AGI. That's a lot to think about, thanks for explaining it to me.