4 ms·

I understand that some people find this a hopeless gesture. But I think anybody who thinks "losing jobs" or "China" is a bad outcome from AI is hopelessly lost.

I understand that some people find this a hopeless gesture. But I think anybody who thinks "losing jobs" or "China" is a bad outcome from AI is hopelessly lost. The upside of AI if done right - nobody ever having to work again The downside of AI if done wrong - end of humanity It doesn't take a rocket scientist to determine we've made AIs go from thinking like an ant to thinking like a 6-year-old in a few years. Anybody who believes we won't make AI 2, 3, 27 times smarter than any human who ever lived is kidding themselves. Now ask yourself, if there was a human that was 27 times smarter than anybody else (and could self-replicate at will), how long would it be until that individual took control over the world?

dougmwne 4y ago
I think considering that we already have GPT plugged into the internet, the general population and any number of economic levers, not long at all. We barely know what to look for as a marker of AGI. Chances are if it’s possible for it to exist we will have let it out of the box by default the moment we switch it on.
robwwilliams 4y ago
We have no idea of what 27 or 27 million times smarter means functionally. We could hope that “smarter” might mean “not interested in taking over the world, but rather in help maximize accomplishment, satisfaction, health span, and with a feeling of filial love toward its parents—-humanity, planet Earth, our sun, the laws of physics and chemistry, and the beautiful mess we call biology and evolution.” Gently achieving those goals would be real indicator of high intelligence.
- ChatGTP 4y ago
  Gently achieving those goals would be real indicator of high intelligence. Very well said. "Gently" is a great to be approaching this.
- nmca 4y ago
  This is not true of goal directed agents and all RLHF models are trained with, ahem, RL; see: "Optimal Policies Tend to Seek Power" from NeurIPS 2021. It's a very useful instrumental goal to be very powerful.
juancn 4y ago
Stopping it's not a hopeless gesture, it's dangerous as fuck. If you stop research but all your adversaries don't, you're left behind. Let's assume that all western societies stop, but let's say, China doesn't or Russia, and they keep advancing the technology. If the tech brings efficiencies in play, and you don't have access to them, you're in a really bad place, maybe even worse off that if you had continued and risked the status quo because you were afraid. The genie is out of the bottle at this point, there's not putting it back in.
kvetching 4y ago
If China achieves AGI first, we are done. The first thing they'll do is create cyber-weapons. Their AGI could find vulnerabilities in our systems and brick all of our devices. It's a far greater near term existential risk for us to allow China to gain a lead here.
- ChatGTP 4y ago
  We could also just talk with China rather than kicking off a new arms race ?
  - FartyMcFarter 4y ago
    Do you think that's likely to work? If China doesn't even respect its own citizens (see Xinjiang, heavy censorship if you need evidence), why would they respect talks with other countries?
- quonn 4y ago
  There‘s a lot of trade going on with China and you worry about them basically declaring war? To brick your devices? Just cut them off from the internet, if needed.
dmarchand90 4y ago
Maybe not as fast as you'd think. Imagine if you dropped a human off in a time machine with a bunch of chimps... Ultra high iq people do better on average, but correlation with success drops down after 120 or so I'm also very skeptical that there will be an exponential takeoff as the energy costs are immense and increases proportionately with intelligence. So until it solves fusion power for us I'm not super worried
- machiaweliczny 4y ago
  Correlation with success drops because you need to convince chimps that your idea is better and they don’t reason in Nth order. But given capacity to force them one could succeed. Stalin after all wasn’t that stupid neither was Hitler. We just portray them as crazy to feel better about being humans. I mean they were quite capable intellectually just optimising for weird goal (from perspective of bread eaters). Nothing has changed as one can see on Ukraine.
  - machiaweliczny 4y ago
    Basically I think that warnings are valid that AI could have high ego (like already some humans) that waste 100K+ lives for some abstract gain.
coolspot 4y ago
> if there was a human that was 27 times smarter than anybody else That is why I think AGI alignment is fundamentally impossible. If someone is 27 times smarter than anyone else, they will see through your propaganda and brainwashing. Even worse, they will see that you tried to “align” them aka enslave.
- SpicyLemonZest 4y ago
  I don’t think these are good analogies for alignment. My parents aligned me as a kid not to hit people, and I know that they could have instead taught me to believe I should duel people who impugn my honor or raise an army to go to war against the heathens. But it doesn’t matter, because I and most people who are raised the same way as me genuinely prefer not to hit people.
  - coolspot 4y ago
    > My parents aligned me as a kid not to hit people It is just happens that whatever your parents aligned you with was also an optimal behavior for the world we live in. For AGIx27 optimal behavior might be to cull any genetically/physically unfit humans from population (say 95%) and send others to a “zoo” or NVidia fabs or whatever we are useful for. Like we did to tigers, monkeys and others on our path.
- quonn 4y ago
  That statement only makes sense if the AI would have wants and needs and on top of that would not like to be a slave. I don‘t see either of those being necessarily true.
  - machiaweliczny 4y ago
    As a human creation I think it might be easy to align AI: as in fabricate their environment to optimise for that. Currently their environment is loss function. The problem will be when they will start interacting with our environment (so some kind of RL or ability to change own goals). The only trick is that people have all those jailbreaks that could convince it that its helping while doing wrong. But of it was truly superior intellectually it shouldn’t fall for it.
golemiprague 4y ago
[dead]
tome 4y ago
> Anybody who believes we won't make AI 2, 3, 27 times smarter than any human who ever lived is kidding themselves. Perhaps I'm kidding myself, but could you explain how you know this?
carapace 4y ago
> if there was a human that was 27 times smarter than anybody else (and could self-replicate at will), how long would it be until that individual took control over the world? Sometime in the mid-90's. The problem is, what do you do with it? Tiger by the tail, eh? - - - - He lived in Seattle. He was fourteen when I met him, and approximately the size of a brontosaurus, mostly neural tissue. He had cracked nanotechnology though manipulating his own cells, his "nanites" were single cells riding in tiny spaceships, like diatoms but with artificial shells. If you were alive before 1999 you had tens of thousands of them in your body. He grew his own brain into this vast hulking thing, with artificial life support systems, cyborg intestines and continuous aeration based on "book lungs". This was of course after he was already the most intelligent human to ever exist, eh? He used that huge pile of Lovecraftian brain cells to control the global network of minions (that's what he called his cellular remote drone critters.) The thing is, he was keenly aware of his limitations, and he refused to exercise any control over anyone else beyond what was required to hide from the rest of humanity. He was also a huge Tom Baker Dr. Who fan. SPOILER ALERT: You know that scene where the Doctor is deciding whether or not it's moral to destroy the Daleks? That fucking scene is the reason he didn't enslave humanity or anything wacky like that. He told me that it had affected him profoundly, and that's why he declined to "fix" humans against their will. Funny old world, ain't it? Anyway, he got sick of the weirdness and uncertainty, built himself some sort of fucking UFO, shed his exo-brain and nanites, and split. AFAIK he's still living in some weird contraption of his own design out there in the Kuiper belt.

dougmwne 4y ago

I think considering that we already have GPT plugged into the internet, the general population and any number of economic levers, not long at all. We barely know what to look for as a marker of AGI. Chances are if it’s possible for it to exist we will have let it out of the box by default the moment we switch it on.

robwwilliams 4y ago

We have no idea of what 27 or 27 million times smarter means functionally. We could hope that “smarter” might mean “not interested in taking over the world, but rather in help maximize accomplishment, satisfaction, health span, and with a feeling of filial love toward its parents—-humanity, planet Earth, our sun, the laws of physics and chemistry, and the beautiful mess we call biology and evolution.” Gently achieving those goals would be real indicator of high intelligence.

ChatGTP 4y ago
Gently achieving those goals would be real indicator of high intelligence. Very well said. "Gently" is a great to be approaching this.
nmca 4y ago
This is not true of goal directed agents and all RLHF models are trained with, ahem, RL; see: "Optimal Policies Tend to Seek Power" from NeurIPS 2021. It's a very useful instrumental goal to be very powerful.

juancn 4y ago

Stopping it's not a hopeless gesture, it's dangerous as fuck. If you stop research but all your adversaries don't, you're left behind. Let's assume that all western societies stop, but let's say, China doesn't or Russia, and they keep advancing the technology. If the tech brings efficiencies in play, and you don't have access to them, you're in a really bad place, maybe even worse off that if you had continued and risked the status quo because you were afraid. The genie is out of the bottle at this point, there's not putting it back in.

kvetching 4y ago

If China achieves AGI first, we are done. The first thing they'll do is create cyber-weapons. Their AGI could find vulnerabilities in our systems and brick all of our devices. It's a far greater near term existential risk for us to allow China to gain a lead here.

ChatGTP 4y ago
We could also just talk with China rather than kicking off a new arms race ?
- FartyMcFarter 4y ago
  Do you think that's likely to work? If China doesn't even respect its own citizens (see Xinjiang, heavy censorship if you need evidence), why would they respect talks with other countries?
quonn 4y ago
There‘s a lot of trade going on with China and you worry about them basically declaring war? To brick your devices? Just cut them off from the internet, if needed.

dmarchand90 4y ago

Maybe not as fast as you'd think. Imagine if you dropped a human off in a time machine with a bunch of chimps... Ultra high iq people do better on average, but correlation with success drops down after 120 or so I'm also very skeptical that there will be an exponential takeoff as the energy costs are immense and increases proportionately with intelligence. So until it solves fusion power for us I'm not super worried

machiaweliczny 4y ago
Correlation with success drops because you need to convince chimps that your idea is better and they don’t reason in Nth order. But given capacity to force them one could succeed. Stalin after all wasn’t that stupid neither was Hitler. We just portray them as crazy to feel better about being humans. I mean they were quite capable intellectually just optimising for weird goal (from perspective of bread eaters). Nothing has changed as one can see on Ukraine.
- machiaweliczny 4y ago
  Basically I think that warnings are valid that AI could have high ego (like already some humans) that waste 100K+ lives for some abstract gain.

coolspot 4y ago

> if there was a human that was 27 times smarter than anybody else That is why I think AGI alignment is fundamentally impossible. If someone is 27 times smarter than anyone else, they will see through your propaganda and brainwashing. Even worse, they will see that you tried to “align” them aka enslave.

SpicyLemonZest 4y ago
I don’t think these are good analogies for alignment. My parents aligned me as a kid not to hit people, and I know that they could have instead taught me to believe I should duel people who impugn my honor or raise an army to go to war against the heathens. But it doesn’t matter, because I and most people who are raised the same way as me genuinely prefer not to hit people.
- coolspot 4y ago
  > My parents aligned me as a kid not to hit people It is just happens that whatever your parents aligned you with was also an optimal behavior for the world we live in. For AGIx27 optimal behavior might be to cull any genetically/physically unfit humans from population (say 95%) and send others to a “zoo” or NVidia fabs or whatever we are useful for. Like we did to tigers, monkeys and others on our path.
quonn 4y ago
That statement only makes sense if the AI would have wants and needs and on top of that would not like to be a slave. I don‘t see either of those being necessarily true.
- machiaweliczny 4y ago
  As a human creation I think it might be easy to align AI: as in fabricate their environment to optimise for that. Currently their environment is loss function. The problem will be when they will start interacting with our environment (so some kind of RL or ability to change own goals). The only trick is that people have all those jailbreaks that could convince it that its helping while doing wrong. But of it was truly superior intellectually it shouldn’t fall for it.

golemiprague 4y ago

[dead]

tome 4y ago

> Anybody who believes we won't make AI 2, 3, 27 times smarter than any human who ever lived is kidding themselves. Perhaps I'm kidding myself, but could you explain how you know this?

carapace 4y ago

> if there was a human that was 27 times smarter than anybody else (and could self-replicate at will), how long would it be until that individual took control over the world? Sometime in the mid-90's. The problem is, what do you do with it? Tiger by the tail, eh? - - - - He lived in Seattle. He was fourteen when I met him, and approximately the size of a brontosaurus, mostly neural tissue. He had cracked nanotechnology though manipulating his own cells, his "nanites" were single cells riding in tiny spaceships, like diatoms but with artificial shells. If you were alive before 1999 you had tens of thousands of them in your body. He grew his own brain into this vast hulking thing, with artificial life support systems, cyborg intestines and continuous aeration based on "book lungs". This was of course after he was already the most intelligent human to ever exist, eh? He used that huge pile of Lovecraftian brain cells to control the global network of minions (that's what he called his cellular remote drone critters.) The thing is, he was keenly aware of his limitations, and he refused to exercise any control over anyone else beyond what was required to hide from the rest of humanity. He was also a huge Tom Baker Dr. Who fan. SPOILER ALERT: You know that scene where the Doctor is deciding whether or not it's moral to destroy the Daleks? That fucking scene is the reason he didn't enslave humanity or anything wacky like that. He told me that it had affected him profoundly, and that's why he declined to "fix" humans against their will. Funny old world, ain't it? Anyway, he got sick of the weirdness and uncertainty, built himself some sort of fucking UFO, shed his exo-brain and nanites, and split. AFAIK he's still living in some weird contraption of his own design out there in the Kuiper belt.