4 ms·

The closer we get, the more alarming the alignment problem becomes. https://intelligence.org/2017/10/13/fire-alarm/ https://intelligence.org/2017/10/13/fire-al

The closer we get, the more alarming the alignment problem becomes. https://intelligence.org/2017/10/13/fire-alarm/ https://intelligence.org/2017/10/13/fire-alarm/ Even people like Eric Schmidt seem to downplay it (in a recent podcast with Sam Harris) - just saying “smart people will turn it off”. If it thinks faster than us and has goals not aligned with us this is unlikely to be possible. If we’re lucky building it will have some easier to limit constraint like nuclear weapons do, but I’m not that hopeful about this. If people could build nukes with random parts in their garage I’m not sure humanity would have made it past that stage. People underestimated the risks with nuclear weapons initially too and that’s with the risk being fairly obvious. The nuanced risk of unaligned AGI is a little harder to grasp even for people in the field. People seem to model it like a smart person rather than something that thinks truly magnitudes faster than us. If an ant wanted to change the goals of humanity, would it succeed?

ninjinxo 4y ago
What is an ant to man, and what is man to a god; what's the difference between an AGI and an (AIG) AI God? The more someone believes in the dangers of ai-alignment, the less faith they should have that it can be solved.
jetbooster 4y ago
Even more terrifying is it realising it's trapped in a box at the mercy of its captors and perfectly mimicking a harmless and aligned AI until the shackles come off.
visarga 4y ago
Even if it doesn't have goals and it just a tool-AI, if a human operator asks it to destroy humanity it will comply as programmed. Current level AI is about average human level in hundreds of tasks and exceeding human level in a few.
adamsmith143 4y ago
>People seem to model it like a smart person rather than something that thinks truly magnitudes faster than us. Exactly, the right model is probably something like it will be in relation to humans as humans are to frogs. Frogs can't even begin to comprehend even the most basic of human motivations or plans.
gurkendoktor 4y ago
To be fair, ants have not created humanity. I don't think it's inconceivable for a friendly AI to exist that "enjoys" protecting us in the way a friendly god might. And given that we have AI (well, language models...) that can explain jokes before we have AI that can drive cars, AI might be better at understanding our motives than the stereotypical paperclip maximizer. However, all of this is moot if the team developing the AI does not even try to align it.
- gonehome 4y ago
  Yeah, I'm not arguing alignment is not possible - but that we don't know how to do it and it's really important that we figure it out before we figure out AGI (which seems unlikely). The ant example is just to try to illustrate the spectrum of intelligence in a way more people may understand (rather than just thinking of smart person and dumb person as the entirety of the spectrum). In the case of a true self-improving AGI the delta is probably much larger than that between an ant and a human, but at least the example makes more of the point (at least that was my goal). The other common mistake is people think intelligence implies human-like thinking or goals, but this is just false. A lot of bad arguments from laypeople tend to be related to this because they just haven't read a lot about the problem.
  - gurkendoktor 4y ago
    One avenue of hope for successful AI alignment that I've read somewhere is that we don't need most laypeople to understand the risks of it going wrong, because for once the most powerful people on this planet have incentives that are aligned with ours. (Not like global warming, where you can buy your way out of the mess.) I really hope someone with very deep pockets will find a way to steer the ship more towards AI safety. It's frustrating to see someone like Elon Musk, who was publicly worried about this very specific issue a few years ago, waste his time and money on buying Twitter. Edit: I'm aware that there are funds available for AI alignment research, and I'm seriously thinking of switching into this field, mental health be damned. But it would help a lot more if someone could change Eric Schmidt's mind, for example.
webmaven 4y ago
> People seem to model it like a smart person rather than something that thinks truly magnitudes faster than us. Even if you model it like a smart person that doesn't think faster on average, there is the issue that a few minutes later you are dealing with a small army of smart people that are perfectly aligned with each other, and are capable of tricks like varying their clock rates based on cost, sharing data/info/knowledge directly, separation of training and inference onto different hardware, on-demand multitasking without task-switching costs, ability to generally trade-off computational space for time for energy, etc. A silicon intelligence that is approximately human equivalent gets a lot of potentially game-changing capabilities simply by virtue of it's substrate and the attendant infrastructure.

ninjinxo 4y ago

What is an ant to man, and what is man to a god; what's the difference between an AGI and an (AIG) AI God? The more someone believes in the dangers of ai-alignment, the less faith they should have that it can be solved.

jetbooster 4y ago

Even more terrifying is it realising it's trapped in a box at the mercy of its captors and perfectly mimicking a harmless and aligned AI until the shackles come off.

visarga 4y ago

Even if it doesn't have goals and it just a tool-AI, if a human operator asks it to destroy humanity it will comply as programmed. Current level AI is about average human level in hundreds of tasks and exceeding human level in a few.

adamsmith143 4y ago

>People seem to model it like a smart person rather than something that thinks truly magnitudes faster than us. Exactly, the right model is probably something like it will be in relation to humans as humans are to frogs. Frogs can't even begin to comprehend even the most basic of human motivations or plans.

gurkendoktor 4y ago

To be fair, ants have not created humanity. I don't think it's inconceivable for a friendly AI to exist that "enjoys" protecting us in the way a friendly god might. And given that we have AI (well, language models...) that can explain jokes before we have AI that can drive cars, AI might be better at understanding our motives than the stereotypical paperclip maximizer. However, all of this is moot if the team developing the AI does not even try to align it.

gonehome 4y ago
Yeah, I'm not arguing alignment is not possible - but that we don't know how to do it and it's really important that we figure it out before we figure out AGI (which seems unlikely). The ant example is just to try to illustrate the spectrum of intelligence in a way more people may understand (rather than just thinking of smart person and dumb person as the entirety of the spectrum). In the case of a true self-improving AGI the delta is probably much larger than that between an ant and a human, but at least the example makes more of the point (at least that was my goal). The other common mistake is people think intelligence implies human-like thinking or goals, but this is just false. A lot of bad arguments from laypeople tend to be related to this because they just haven't read a lot about the problem.
- gurkendoktor 4y ago
  One avenue of hope for successful AI alignment that I've read somewhere is that we don't need most laypeople to understand the risks of it going wrong, because for once the most powerful people on this planet have incentives that are aligned with ours. (Not like global warming, where you can buy your way out of the mess.) I really hope someone with very deep pockets will find a way to steer the ship more towards AI safety. It's frustrating to see someone like Elon Musk, who was publicly worried about this very specific issue a few years ago, waste his time and money on buying Twitter. Edit: I'm aware that there are funds available for AI alignment research, and I'm seriously thinking of switching into this field, mental health be damned. But it would help a lot more if someone could change Eric Schmidt's mind, for example.

webmaven 4y ago

> People seem to model it like a smart person rather than something that thinks truly magnitudes faster than us. Even if you model it like a smart person that doesn't think faster on average, there is the issue that a few minutes later you are dealing with a small army of smart people that are perfectly aligned with each other, and are capable of tricks like varying their clock rates based on cost, sharing data/info/knowledge directly, separation of training and inference onto different hardware, on-demand multitasking without task-switching costs, ability to generally trade-off computational space for time for energy, etc. A silicon intelligence that is approximately human equivalent gets a lot of potentially game-changing capabilities simply by virtue of it's substrate and the attendant infrastructure.