3 ms·

AGI isn't scary because it's smarter than me. AGI is scary because it's alien. I've met plenty of people smarter than me. Way way smarter. It's ok. We can sorta

by jfoutz 4y ago

AGI isn't scary because it's smarter than me. AGI is scary because it's alien. I've met plenty of people smarter than me. Way way smarter. It's ok. We can sorta work out what the other wants. We can talk or fight or trade or whatever. I have an uncountable number of assumptions to get by in the world. I have this vast heritage of knowledge. I try to keep a handle on why I believe what I believe, but it's not easy, and I'm sure I slip up. An alien would share none of these. Take counting. That's super important for lots of stuff I do every day, like, what day it is. the https://en.wikipedia.org/wiki/Munduruku https://en.wikipedia.org/wiki/Munduruku people don't count. I'm sure we could work something out - but we all gotta sleep and eat and stuff so we can sorta work out a framework for understanding wants and needs and such. An alien doesn't require wants or needs. they may do stuff that we label wants and needs, and it might be a good mapping, but who fucking knows? Aliens are alien. The water is triangular sentence is a great one. do the aliens mean they know that oxygen and 2 hydrogens don't make a straight line? And they know we mean oxygen and two hydrogens are water? and they know 3 non parallel lines in a plane means triangle? Maybe? Sure? maybe they mean all Ferrari's will be consumed for fuel. Mean, dear reader, doesn't even mean anything, because they're aliens. I mean, you can point at math, prime numbers are a great one! but like, do you need prime numbers? Who knows? That's the path we took, but what would an alien need? I don't know. You, dear reader, may have strong opinions, but I'm pretty sure you can offer no proof.

Misdicorl 4y ago
This feels a bit too dismissive of human capability to me. In the end all that "work something out" is emulating the counterparty in your own head given their actions and statements. That emulation will certainly be very difficult when the counterparty is running on different hardware, but it's not impossible imo. And humans get to back propagate error into their emulation model too... Find the pattern is still the paramount skill, something humans are good at, and will continue to be useful when the counterparty is an AI
- jfoutz 4y ago
  Oh for sure! Humans are smart. Collectively we figure stuff out. I think a reasonable example might be mountain lions. There are specific humans that know lots about mountain lions. Generally people know about mountain lions, and to avoid them. But every year a few hikers get picked off. If it's bad enough, we go kill the mountain lion. A mountain lion isn't AGI. It isn't alien. I believe people would agree they're cunning. But there's no treaty to sign to protect humans from mountain lions. There's no counterparty to discuss things with. We, humans, get great advantages from working collaboratively and collectively. I hope we can build good models, I hope there is opportunity for working with an AGI counterparty. But it's not a given. it's an alien. and all of that work _may_ have to start from scratch. We can't know what we have to work with till it's here. Maybe it'll be very easy. And that would be great. That's not a given. To be clear, I'm a wide eyed optimist. it would be fantastic to not be alone, the dream of Star Trek or Ian Banks's Culture would be awesome. But, it might not work out like that. I have high hopes. But they're just hopes.
WhitneyLand 4y ago
Your concern of alien to us sounds similar to the concerns of alignment, which is a primary area of research: https://en.wikipedia.org/wiki/AI_alignment https://en.wikipedia.org/wiki/AI_alignment
- jfoutz 4y ago
  Yeah, that's exactly it. Talk about a great example of me failing to know why I know things. I think I encountered that on Rational wiki years ago. I subsequently "smushed" that idea into a bunch of science fiction, Frankenstein's monster and the aliens in Ender's game have a similar flavor - in my mind anyway. It's great that's an active area of research. from the outside, it seems extraordinarily difficult. Persuading a person from a similar cultural background is tough enough. I can't imagine the challenges of persuading an AI. I realize I'm taking "persuasion" and "agreement" as a given here, and those are ideas we might not share with an AGI. Seems real hard.

by jfoutz 4y ago

Misdicorl 4y ago

This feels a bit too dismissive of human capability to me. In the end all that "work something out" is emulating the counterparty in your own head given their actions and statements. That emulation will certainly be very difficult when the counterparty is running on different hardware, but it's not impossible imo. And humans get to back propagate error into their emulation model too... Find the pattern is still the paramount skill, something humans are good at, and will continue to be useful when the counterparty is an AI

jfoutz 4y ago
Oh for sure! Humans are smart. Collectively we figure stuff out. I think a reasonable example might be mountain lions. There are specific humans that know lots about mountain lions. Generally people know about mountain lions, and to avoid them. But every year a few hikers get picked off. If it's bad enough, we go kill the mountain lion. A mountain lion isn't AGI. It isn't alien. I believe people would agree they're cunning. But there's no treaty to sign to protect humans from mountain lions. There's no counterparty to discuss things with. We, humans, get great advantages from working collaboratively and collectively. I hope we can build good models, I hope there is opportunity for working with an AGI counterparty. But it's not a given. it's an alien. and all of that work _may_ have to start from scratch. We can't know what we have to work with till it's here. Maybe it'll be very easy. And that would be great. That's not a given. To be clear, I'm a wide eyed optimist. it would be fantastic to not be alone, the dream of Star Trek or Ian Banks's Culture would be awesome. But, it might not work out like that. I have high hopes. But they're just hopes.

WhitneyLand 4y ago

Your concern of alien to us sounds similar to the concerns of alignment, which is a primary area of research: https://en.wikipedia.org/wiki/AI_alignment https://en.wikipedia.org/wiki/AI_alignment

jfoutz 4y ago
Yeah, that's exactly it. Talk about a great example of me failing to know why I know things. I think I encountered that on Rational wiki years ago. I subsequently "smushed" that idea into a bunch of science fiction, Frankenstein's monster and the aliens in Ender's game have a similar flavor - in my mind anyway. It's great that's an active area of research. from the outside, it seems extraordinarily difficult. Persuading a person from a similar cultural background is tough enough. I can't imagine the challenges of persuading an AI. I realize I'm taking "persuasion" and "agreement" as a given here, and those are ideas we might not share with an AGI. Seems real hard.