3 ms·

This feels a bit too dismissive of human capability to me. In the end all that "work something out" is emulating the counterparty in your own head given their

This feels a bit too dismissive of human capability to me. In the end all that "work something out" is emulating the counterparty in your own head given their actions and statements. That emulation will certainly be very difficult when the counterparty is running on different hardware, but it's not impossible imo. And humans get to back propagate error into their emulation model too... Find the pattern is still the paramount skill, something humans are good at, and will continue to be useful when the counterparty is an AI

jfoutz 4y ago
Oh for sure! Humans are smart. Collectively we figure stuff out. I think a reasonable example might be mountain lions. There are specific humans that know lots about mountain lions. Generally people know about mountain lions, and to avoid them. But every year a few hikers get picked off. If it's bad enough, we go kill the mountain lion. A mountain lion isn't AGI. It isn't alien. I believe people would agree they're cunning. But there's no treaty to sign to protect humans from mountain lions. There's no counterparty to discuss things with. We, humans, get great advantages from working collaboratively and collectively. I hope we can build good models, I hope there is opportunity for working with an AGI counterparty. But it's not a given. it's an alien. and all of that work _may_ have to start from scratch. We can't know what we have to work with till it's here. Maybe it'll be very easy. And that would be great. That's not a given. To be clear, I'm a wide eyed optimist. it would be fantastic to not be alone, the dream of Star Trek or Ian Banks's Culture would be awesome. But, it might not work out like that. I have high hopes. But they're just hopes.