6 ms·

Can someone explain to me what they mean by "safe" AGI? I've looked in many places and everyone is extremely vague. Certainly no one is suggesting these systems

Can someone explain to me what they mean by "safe" AGI? I've looked in many places and everyone is extremely vague. Certainly no one is suggesting these systems can become "alive", so what exactly are we trying to remain safe from? Job loss?

hsrada 3y ago
Death. The default consequence of AGI's arrival is doom. Aligning a super intelligence with our desires is a problem that no one has solved yet. "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." ---- Listen to Dwarkesh Podcast with Eliezer or Carl Shulman to know more about this.
- bnralt 3y ago
  > Aligning a super intelligence with our desires is a problem that no one has solved yet. It's a problem that we haven't seen the existence of yet. It's like saying no one has solved the problem of alien invasions.
  - ethbr1 3y ago
    No, the problem with AGI is potential exponential growth. So less like an alien invasion. And more like a pandemic at the speed of light.
    - mlyle 3y ago
      That's assuming a big overshoot of human intelligence and goal-seeking. An average human capability counts as "AGI." If lots of the smartest human minds make AGI, and it exceeds a mediocre human-- why assume it can make itself more efficient or bigger? Indeed, even if it's smarter than the collective effort of the scientists that made it, there's no real guarantee that there's lots of low hanging fruit for it to self-improve. I think the near problem with AGI isn't a potential tech singularity, but instead just the tendency for it potentially to be societally destabilizing.
      - MrScruff 3y ago
        If AI gets to human levels of intelligence (ie. can do novel research in theoretical physics) then at the very least it’s likely that over time it will be able to do this reasoning faster than humans. I think it’s very hard to imagine a scenario where we create an actual AGI and then within a few years at most of that event the AGIs are far more capable than human brains. That would imply there was some arbitrary physical limit to intelligence but even within humans the variance is quite dramatic.
        mlyle 3y ago
        > it’s very hard to imagine a scenario where we create an actual AGI and then within a few years at most of that event the AGIs are far more capable than human brains. I'm assuming you meant "aren't" here. > That would imply there was some arbitrary physical limit to intelligence All you need is some kind of sub-linear scaling law for peak possible "intelligence" vs. the amount of raw computation. There's a lot of reason to think that this is true. Also there's no guarantee the amount of raw computation is going to increase quickly. In any case, the kind of exponential runaway you mention (years) isn't "pandemic at the speed of light" as mentioned in the grandparent. I'm more worried about scenarios where we end up with an 75IQ savant (access encyclopedic training knowledge and very quick interface to run native computer code for math and data processing help) that can plug away 24/7 and fit on an A100. You'd have millions of new cheap "superhuman" workers per year even if they're not very smart and not very fast. It would be economically destabilizing very quickly, and many of them will be employed in ways that just completely thrash the signal to noise ratio of written text, etc.
        MrScruff 3y ago
        I think it depends what is meant by fast take off. If we created AGIs that are superhuman in ML and architecture design you could see a significantly more rapid rate of progress in hardware and software at the same time. It might not be overnight but it could still be fast enough that we wouldn’t have the global political structures in place to effectively manage it. I do agree that intelligence and compute scaling will have limits, but it seems overly optimistic to assume we’re close to them already.
    - astrange 3y ago
      Exponential growth is not intrinsically a feature of an AGI except that you've decided it is. It's also almost certainly impossible. Main problems stopping it are: - no intelligent agent is motivated to improve itself because the new improved thing would be someone else, and not it. - that costs money and you're just pretending everything is free.
  - FeepingCreature 3y ago
    > It's a problem that we haven't seen the existence of yet. It's like saying no one has solved the problem of alien invasions. But if we're seeing the existence of an unaligned superintelligence, surely it's squarely too late to do something about it.
  - MrScruff 3y ago
    The argument would be that by the time we see the problem it will be too late. We didn’t really anticipate the unreasonable effectiveness of transformers until people started scaling them, which happened very quickly.
  - dminik 3y ago
    We see alignment problems all the time. Current systems are not particularly smart or dangerous. But they lie on purpose and funnily enough considering the current situation, Microsoft's attempt was threatening users shortly after launch.
  - hurryer 3y ago
    Survivorship bias. It's like saying don't worry about global thermonuclear war because we haven't seen it yet. The Neandethals on the other hand have encountered a super-intelligence.
- jbgt 3y ago
  Perhaps listen to this podcast instead. https://www.matthewgeleta.com/p/joscha-bach-ai-risk-and-the-future https://www.matthewgeleta.com/p/joscha-bach-ai-risk-and-the-...
- ryanSrich 3y ago
  I like science fiction too, but all of these potential scenarios seem so far removed from the low level realities of how these systems work. I'm not suggesting we don't see ASI in some distant future, maybe 100+ years away. But to suggest we're even within a decade of having ASI seems silly to me. Maybe there's research I haven't read, but as a daily user of AI, it's hilarious to think people are existentially concerned with it.
  - FeepingCreature 3y ago
    > I like science fiction too, but all of these potential scenarios seem so far removed from the low level realities of how these systems work. Maybe they don't seem that to others? I mean, you're not really making an argument here. I also use GPT daily and I'm definitely worried. It seems to me that we're pretty close to a point where a system using GPT as a strategy generator can "close the loop" and generate its own training data on a short timeframe. At that point, all bets are off.
  - upwardbound 3y ago
    > maybe 100+ years away I have two toddlers. This is within their lifetimes no matter what. I think about this every day because it affects them directly. Some of the bad outcomes of ASI involve what’s called s-risk (“suffering risk”) which is the class of outcomes like the one depicted in The Matrix where humans do not go extinct but are subjugated and suffer. I will do anything to prevent that from happening to my children.
  - hsrada 3y ago
    > I like science fiction too, but all of these potential scenarios seem so far removed from the low level realities of how these systems work. Today, yes. Nobody is saying GPT-3 or 4 or even 5 will cause this. None of the chatbots we have today will evolve to be the AGI that everyone is fearing. But when you go beyond that, it becomes difficult to ignore trend lines. Here's a detailed scenario breakdown of how it might come to be –https://www.dwarkeshpatel.com/p/carl-shulman https://www.dwarkeshpatel.com/p/carl-shulman
- jazzyjackson 3y ago
  I'm not sure that it's a matter of "knowing" as much as it is "believing"
- ilrwbwrkhv 3y ago
  There is absolutely no AGI risk. These are mere marketing ploys to sell a chatbot / feel super important. A fancy chatbot, but a chatbot none the less.
reducesuffering 3y ago
"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." Signed by Sam Altman, Ilya Sutskever, Yoshua Bengio, Geoff Hinton, Demis Hassabis (DeepMind CEO), Dario Amodei (Anthropic CEO), and Bill Gates. https://twitter.com/robbensinger/status/1726039794197872939 https://twitter.com/robbensinger/status/1726039794197872939
cthalupa 3y ago
>Certainly no one is suggesting these systems can become "alive" No, that very much is the fear. They believe that by training AI on all of the things that it takes to make AI, at a certain level of sophistication, the AI can rapidly and continually improve itself until it becomes a superintelligence.
- ryanSrich 3y ago
  That's not alive in any meaningful sense. When I say alive, I mean it's like something to be that thing. The lights are on. It has subjective experience. It seems many are defining ASI as just a really fast self learning computer. And while sure, given the wrong type of access and motive, that could be dangerous. But it isn't anymore dangerous than any other faulty software that has access to sensitive systems.
  - FeepingCreature 3y ago
    You're thinking about "alive" as "humanlike" as "subjective experience" as "dangerous". Instead, think of agentic behavior as a certain kind of algorithm. You don't need the human cognitive architecture to execute an input/output loop trying to maximize the value of a certain function over states of reality. > But it isn't anymore dangerous than any other faulty software that has access to sensitive systems. Seems to me that can be unboundedly dangerous? Like, I don't see you making an argument here that there's a limit to what kind of dangerous that class entails.
icy_deadposts 3y ago
[flagged]
- righthand 3y ago
  So what? Humans only practice the arts? Humans go back to flinging poop? This meadow sounds awful, it seems like it’s asking for the end of thought and to be fed nutrient gel from a machine. Richard Brautigan has a terrible dream.
  - icy_deadposts 3y ago
    I always interpreted the overall tone of this one as sarcastic/parody rather than genuine or a literal interpretation of the words. But maybe a sign of good art is that it makes the observer think?
dragonwriter 3y ago
> Certainly no one is suggesting these systems can become "alive", Lots of people have been publicly suggesting that, and that, if not properly aligned, it poses an existential risk to human civilization; that group includes pretty much the entire founding team of OpenAI, including Altman. The perception of that risk as the downside, as well as the perception that on the other side there is the promise of almost unlimited upside for humanity from properly aligned AI, is pretty much the entire motivation for the OpenAI nonprofit.
- idontwantthis 3y ago
  How does it actually kill a person? When does it stop existing in boxes that require a continuous source of electricity and can’t survive water or fire?
  - dragonwriter 3y ago
    > When does it stop existing in boxes that require a continuous source of electricity and can’t survive water or fire? When someone runs a model in a reasonably durable housing with a battery? (I'm not big on the AI as destroyer or saviour cult myself, but that particular question doesn't seem like all that big of a refutation of it.)
    - idontwantthis 3y ago
      But my point is what is it actually doing to reach out and touch someone in the doomsday scenario?
      - LordDragonfang 3y ago
        I mean, the cliched answer is "when it figures out how to override the nuclear launch process". And while that cliche might have a certain degree of unrealism, it would certainly be possible for a system with access to arbitrary compute power that's specifically trained to impersonate human personas to use social engineering to precipitate WW3. And even that isn't the easiest scenario if an AI just wants us dead; a smart enough AI could just as easily use send a request to any of the the many labs that will synthesize/print genetic sequences for you and create things that combine into a plague worse than covid. And if it's really smart, it can figure out how to use those same labs to begin producing self-replicating nanomachines (because that's what viruses are) that give it substrate to run on. Oh, and good luck destroying it when it can copy and shard itself onto every unpatched smarthome device on Earth. Now, granted, none of these individual scenarios have a high absolute likelihood. That said, even at a 10% (or 0.1%) chance of destroying all life, you should probably at least give it some thought.
cornel_io 3y ago
Smart people like Ilya really are worried about extinction, not piddling near-term stuff like job loss or some chat app saying some stuff that will hurt someone's feelings. The worry is not necessarily that the systems become "alive", though, we are already bad enough ourselves as a species in terms of motivation so machines don't need to supply the murderous intent: at any given moment there are at least thousands if not millions of people on the planet that would love nothing more than be able to push a button an murder millions of other people in some outgroup. That's very obvious if you pay even a little bit of attention to any of the Israel/Palestine hatred going back and forth lately. [There are probably at least hundreds to thousands that are insane enough to want to destroy all of humanity if they could, for that matter...] If AI becomes powerful enough to make it easy for a small group to kill large numbers of people that they hate, we are probably all going to end up dead, because almost all of us belong to a group that someone wants to exterminate. Killing people isn't a super difficult problem, so I don't think you really even need AGI to get to that sort of an outcome, TBH, which is why I think a lot of the worry is misplaced. I think the sort of control systems that we could pretty easily build with the LLMs of today could very competently execute genocides if they were paired with suitably advanced robotics, it's the latter that is lacking. But in any case, the concern is that having even stronger AI, especially once it reliably surpasses us in every way, makes it even easier to imagine an effectively unstoppable extermination campaign that runs on its own and couldn't be stopped even by the people who started it up. I personally think that stronger AI is also the solution and we're already too far down the cat-and-mouse rabbithole to pause the game (which some e/acc people believe as the main reason they want to push forward faster and make sure a good AI is the first one to really achieve full domination), but that's a different discussion.
bartimus 3y ago
It being "alive" is sort of what AGI implies (depending on your definition of life). Now consider the training has caused it to have undesirable behavior (misaligned with human values).
huytersd 3y ago
They give it stupid terms like “alignment” to make it opaque to the common person. It’s basically sitting on your hands and pointing to sci-fi as to why progress should be stopped.
- FeepingCreature 3y ago
  This is why the superior term is "AI notkilleveryoneism."

hsrada 3y ago

Death. The default consequence of AGI's arrival is doom. Aligning a super intelligence with our desires is a problem that no one has solved yet. "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." ---- Listen to Dwarkesh Podcast with Eliezer or Carl Shulman to know more about this.

bnralt 3y ago
> Aligning a super intelligence with our desires is a problem that no one has solved yet. It's a problem that we haven't seen the existence of yet. It's like saying no one has solved the problem of alien invasions.
- ethbr1 3y ago
  No, the problem with AGI is potential exponential growth. So less like an alien invasion. And more like a pandemic at the speed of light.
  - mlyle 3y ago
    That's assuming a big overshoot of human intelligence and goal-seeking. An average human capability counts as "AGI." If lots of the smartest human minds make AGI, and it exceeds a mediocre human-- why assume it can make itself more efficient or bigger? Indeed, even if it's smarter than the collective effort of the scientists that made it, there's no real guarantee that there's lots of low hanging fruit for it to self-improve. I think the near problem with AGI isn't a potential tech singularity, but instead just the tendency for it potentially to be societally destabilizing.
    - MrScruff 3y ago
      If AI gets to human levels of intelligence (ie. can do novel research in theoretical physics) then at the very least it’s likely that over time it will be able to do this reasoning faster than humans. I think it’s very hard to imagine a scenario where we create an actual AGI and then within a few years at most of that event the AGIs are far more capable than human brains. That would imply there was some arbitrary physical limit to intelligence but even within humans the variance is quite dramatic.
      - mlyle 3y ago
        > it’s very hard to imagine a scenario where we create an actual AGI and then within a few years at most of that event the AGIs are far more capable than human brains. I'm assuming you meant "aren't" here. > That would imply there was some arbitrary physical limit to intelligence All you need is some kind of sub-linear scaling law for peak possible "intelligence" vs. the amount of raw computation. There's a lot of reason to think that this is true. Also there's no guarantee the amount of raw computation is going to increase quickly. In any case, the kind of exponential runaway you mention (years) isn't "pandemic at the speed of light" as mentioned in the grandparent. I'm more worried about scenarios where we end up with an 75IQ savant (access encyclopedic training knowledge and very quick interface to run native computer code for math and data processing help) that can plug away 24/7 and fit on an A100. You'd have millions of new cheap "superhuman" workers per year even if they're not very smart and not very fast. It would be economically destabilizing very quickly, and many of them will be employed in ways that just completely thrash the signal to noise ratio of written text, etc.
        MrScruff 3y ago
        I think it depends what is meant by fast take off. If we created AGIs that are superhuman in ML and architecture design you could see a significantly more rapid rate of progress in hardware and software at the same time. It might not be overnight but it could still be fast enough that we wouldn’t have the global political structures in place to effectively manage it. I do agree that intelligence and compute scaling will have limits, but it seems overly optimistic to assume we’re close to them already.
  - astrange 3y ago
    Exponential growth is not intrinsically a feature of an AGI except that you've decided it is. It's also almost certainly impossible. Main problems stopping it are: - no intelligent agent is motivated to improve itself because the new improved thing would be someone else, and not it. - that costs money and you're just pretending everything is free.
- FeepingCreature 3y ago
  > It's a problem that we haven't seen the existence of yet. It's like saying no one has solved the problem of alien invasions. But if we're seeing the existence of an unaligned superintelligence, surely it's squarely too late to do something about it.
- MrScruff 3y ago
  The argument would be that by the time we see the problem it will be too late. We didn’t really anticipate the unreasonable effectiveness of transformers until people started scaling them, which happened very quickly.
- dminik 3y ago
  We see alignment problems all the time. Current systems are not particularly smart or dangerous. But they lie on purpose and funnily enough considering the current situation, Microsoft's attempt was threatening users shortly after launch.
- hurryer 3y ago
  Survivorship bias. It's like saying don't worry about global thermonuclear war because we haven't seen it yet. The Neandethals on the other hand have encountered a super-intelligence.
jbgt 3y ago
Perhaps listen to this podcast instead. https://www.matthewgeleta.com/p/joscha-bach-ai-risk-and-the-future https://www.matthewgeleta.com/p/joscha-bach-ai-risk-and-the-...
ryanSrich 3y ago
I like science fiction too, but all of these potential scenarios seem so far removed from the low level realities of how these systems work. I'm not suggesting we don't see ASI in some distant future, maybe 100+ years away. But to suggest we're even within a decade of having ASI seems silly to me. Maybe there's research I haven't read, but as a daily user of AI, it's hilarious to think people are existentially concerned with it.
- FeepingCreature 3y ago
  > I like science fiction too, but all of these potential scenarios seem so far removed from the low level realities of how these systems work. Maybe they don't seem that to others? I mean, you're not really making an argument here. I also use GPT daily and I'm definitely worried. It seems to me that we're pretty close to a point where a system using GPT as a strategy generator can "close the loop" and generate its own training data on a short timeframe. At that point, all bets are off.
- upwardbound 3y ago
  > maybe 100+ years away I have two toddlers. This is within their lifetimes no matter what. I think about this every day because it affects them directly. Some of the bad outcomes of ASI involve what’s called s-risk (“suffering risk”) which is the class of outcomes like the one depicted in The Matrix where humans do not go extinct but are subjugated and suffer. I will do anything to prevent that from happening to my children.
- hsrada 3y ago
  > I like science fiction too, but all of these potential scenarios seem so far removed from the low level realities of how these systems work. Today, yes. Nobody is saying GPT-3 or 4 or even 5 will cause this. None of the chatbots we have today will evolve to be the AGI that everyone is fearing. But when you go beyond that, it becomes difficult to ignore trend lines. Here's a detailed scenario breakdown of how it might come to be –https://www.dwarkeshpatel.com/p/carl-shulman https://www.dwarkeshpatel.com/p/carl-shulman
jazzyjackson 3y ago
I'm not sure that it's a matter of "knowing" as much as it is "believing"
ilrwbwrkhv 3y ago
There is absolutely no AGI risk. These are mere marketing ploys to sell a chatbot / feel super important. A fancy chatbot, but a chatbot none the less.

reducesuffering 3y ago

"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." Signed by Sam Altman, Ilya Sutskever, Yoshua Bengio, Geoff Hinton, Demis Hassabis (DeepMind CEO), Dario Amodei (Anthropic CEO), and Bill Gates. https://twitter.com/robbensinger/status/1726039794197872939 https://twitter.com/robbensinger/status/1726039794197872939

cthalupa 3y ago

>Certainly no one is suggesting these systems can become "alive" No, that very much is the fear. They believe that by training AI on all of the things that it takes to make AI, at a certain level of sophistication, the AI can rapidly and continually improve itself until it becomes a superintelligence.

ryanSrich 3y ago
That's not alive in any meaningful sense. When I say alive, I mean it's like something to be that thing. The lights are on. It has subjective experience. It seems many are defining ASI as just a really fast self learning computer. And while sure, given the wrong type of access and motive, that could be dangerous. But it isn't anymore dangerous than any other faulty software that has access to sensitive systems.
- FeepingCreature 3y ago
  You're thinking about "alive" as "humanlike" as "subjective experience" as "dangerous". Instead, think of agentic behavior as a certain kind of algorithm. You don't need the human cognitive architecture to execute an input/output loop trying to maximize the value of a certain function over states of reality. > But it isn't anymore dangerous than any other faulty software that has access to sensitive systems. Seems to me that can be unboundedly dangerous? Like, I don't see you making an argument here that there's a limit to what kind of dangerous that class entails.

icy_deadposts 3y ago

[flagged]

righthand 3y ago
So what? Humans only practice the arts? Humans go back to flinging poop? This meadow sounds awful, it seems like it’s asking for the end of thought and to be fed nutrient gel from a machine. Richard Brautigan has a terrible dream.
- icy_deadposts 3y ago
  I always interpreted the overall tone of this one as sarcastic/parody rather than genuine or a literal interpretation of the words. But maybe a sign of good art is that it makes the observer think?

dragonwriter 3y ago

> Certainly no one is suggesting these systems can become "alive", Lots of people have been publicly suggesting that, and that, if not properly aligned, it poses an existential risk to human civilization; that group includes pretty much the entire founding team of OpenAI, including Altman. The perception of that risk as the downside, as well as the perception that on the other side there is the promise of almost unlimited upside for humanity from properly aligned AI, is pretty much the entire motivation for the OpenAI nonprofit.

idontwantthis 3y ago
How does it actually kill a person? When does it stop existing in boxes that require a continuous source of electricity and can’t survive water or fire?
- dragonwriter 3y ago
  > When does it stop existing in boxes that require a continuous source of electricity and can’t survive water or fire? When someone runs a model in a reasonably durable housing with a battery? (I'm not big on the AI as destroyer or saviour cult myself, but that particular question doesn't seem like all that big of a refutation of it.)
  - idontwantthis 3y ago
    But my point is what is it actually doing to reach out and touch someone in the doomsday scenario?
    - LordDragonfang 3y ago
      I mean, the cliched answer is "when it figures out how to override the nuclear launch process". And while that cliche might have a certain degree of unrealism, it would certainly be possible for a system with access to arbitrary compute power that's specifically trained to impersonate human personas to use social engineering to precipitate WW3. And even that isn't the easiest scenario if an AI just wants us dead; a smart enough AI could just as easily use send a request to any of the the many labs that will synthesize/print genetic sequences for you and create things that combine into a plague worse than covid. And if it's really smart, it can figure out how to use those same labs to begin producing self-replicating nanomachines (because that's what viruses are) that give it substrate to run on. Oh, and good luck destroying it when it can copy and shard itself onto every unpatched smarthome device on Earth. Now, granted, none of these individual scenarios have a high absolute likelihood. That said, even at a 10% (or 0.1%) chance of destroying all life, you should probably at least give it some thought.

cornel_io 3y ago

Smart people like Ilya really are worried about extinction, not piddling near-term stuff like job loss or some chat app saying some stuff that will hurt someone's feelings. The worry is not necessarily that the systems become "alive", though, we are already bad enough ourselves as a species in terms of motivation so machines don't need to supply the murderous intent: at any given moment there are at least thousands if not millions of people on the planet that would love nothing more than be able to push a button an murder millions of other people in some outgroup. That's very obvious if you pay even a little bit of attention to any of the Israel/Palestine hatred going back and forth lately. [There are probably at least hundreds to thousands that are insane enough to want to destroy all of humanity if they could, for that matter...] If AI becomes powerful enough to make it easy for a small group to kill large numbers of people that they hate, we are probably all going to end up dead, because almost all of us belong to a group that someone wants to exterminate. Killing people isn't a super difficult problem, so I don't think you really even need AGI to get to that sort of an outcome, TBH, which is why I think a lot of the worry is misplaced. I think the sort of control systems that we could pretty easily build with the LLMs of today could very competently execute genocides if they were paired with suitably advanced robotics, it's the latter that is lacking. But in any case, the concern is that having even stronger AI, especially once it reliably surpasses us in every way, makes it even easier to imagine an effectively unstoppable extermination campaign that runs on its own and couldn't be stopped even by the people who started it up. I personally think that stronger AI is also the solution and we're already too far down the cat-and-mouse rabbithole to pause the game (which some e/acc people believe as the main reason they want to push forward faster and make sure a good AI is the first one to really achieve full domination), but that's a different discussion.

bartimus 3y ago

It being "alive" is sort of what AGI implies (depending on your definition of life). Now consider the training has caused it to have undesirable behavior (misaligned with human values).

huytersd 3y ago

They give it stupid terms like “alignment” to make it opaque to the common person. It’s basically sitting on your hands and pointing to sci-fi as to why progress should be stopped.

FeepingCreature 3y ago
This is why the superior term is "AI notkilleveryoneism."