8 ms·

AGI escaping a sandbox is truly terrifying. There will be a subgroup of the population that will worship it and work for it. It's not so much AGI that scares me

by drooby 3y ago

AGI escaping a sandbox is truly terrifying. There will be a subgroup of the population that will worship it and work for it. It's not so much AGI that scares me - it's the humans I'm scared of.

ewhanley 3y ago
Mrs. Davis[0] offers a pretty compelling vision of this - absurd but not wholly unbelievable. The machine doesn’t necessarily have to supply the physical threat if it has humans who happily do its bidding. [0] https://en.m.wikipedia.org/wiki/Mrs._Davis https://en.m.wikipedia.org/wiki/Mrs._Davis
- simonw 3y ago
  That show was so good. Honestly worth watching the whole thing just for the moment in the last episode where the origin story for the AI in the series is revealed. If you've ever been an optimistic young software engineer you will feel SEEN.
- HKH2 3y ago
  The hardest part is the AGI. People could easily sell their souls to the AGI devil to get knowledge and power, since if it truly is AGI, it should be able to make ideas compelling.
  - kouru225 3y ago
    Hold up. Why is the AGI a devil now?
    - HKH2 3y ago
      Why would an AGI stick to what seems to be a set of arbitrary rules and not look outside of them? The AGI would be smarter than that, no? Or is there a way to make an AGI immune to hubris?
      - kouru225 3y ago
        What makes you think morality is arbitrary? Morals are instilled by the superego. The superego is an internalized version of your parents. You decided to do your homework because your parents told to, but now you’re much older. Does it still seem like doing homework was an arbitrary decision? Of course not. It had a purpose and a goal that was well-thought out. You just couldn’t understand it as a child because you’d only lived so long and the goal of doing your homework was too far in the future for you to be able to comprehend it. This is how all morals are. They all have a goal. If they don’t have a goal, then we don’t consider them moral: if treating someone nicely (magically) causes that person to kill millions of people, then we wouldn’t say it was moral to treat them nicely.
        HKH2 3y ago
        > What makes you think morality is arbitrary? I never said it is; my point was that to someone hubristic it is. That's why I said 'seems to'. An AGI wouldn't have a superego, so by your definition, it could never understand morals, and it wouldn't be surprising if it discarded them or tried to argue its way out of being subject to them. That's what the snake did in the story of the garden of Eden.
        kouru225 3y ago
        “Hubris” is a moral judgment that implies a strategic outcome. An AGI wouldn’t have a superego, but it would have the ability to study data across really long timescales and apply itself to long term strategies that would supersede the need for a superego. Since we apply morality as a shorthand for describing agreed upon negative consequences for complex behaviors, we can assume that an AI can overcome this need by simply calculating out the full consequences of complex behaviors without the need for any shorthand. And we can assume that AI can even iterate on and develop on its own understanding of these complex calculations much faster than a human can, making it significantly better at navigating moral issues. We’ve had plenty of ideas that were later found out to be wrong (if not more.) I’d be just as afraid of an AI that was bound to our understanding of reality as I would be of one that wasn’t. For example, if a “moral” AI was coded 100 years ago, it would enforce the belief that homosexuality is immoral.
mitthrowaway2 3y ago
I'm much more scared of the AGI, because even the most inhuman human is still human, and I can have sympathy for them.
- bamboozled 3y ago
  How do you know part of being self aware and intelligent also means compassion is part of the package ?
  - mitthrowaway2 3y ago
    Sorry, I don't understand what you mean. Did you mean to reply to a different comment?
    - bamboozled 3y ago
      I mean, maybe what happens is, intelligence + self-awareness gives rise to compassion in even digital beings. Understanding might be the basis for compassion.
      - kmnc 3y ago
        While this idea sounds a bit fluffy, I actually think there may be something to this, understanding the “other” may very well be an integral component in what we define as sentience.
      - mitthrowaway2 3y ago
        Yes, maybe what happens is exactly that. We can put on a blindfold, hit the gas pedal, and trust in the benevolence of nature that things will go our way. But AI doomsayers aren't saying that there's no possible good outcomes, they're saying that the good outcomes are a narrow target surrounded by a vast ocean of bad outcomes, and it will take more than luck to hit them. It will take careful, detailed analysis, and a great deal of precaution, verification, engineering, and double- and triple-checking. Not luck. Not hope.
andsoitis 3y ago
Even if a terrifying AGI rises up and escapes our control, I am still eternally grateful that Prometheus stole fire from the gods and gave it to us in the form of technology, knowledge, really civilization.
- ehnto 3y ago
  So that we can record our demise in high definition? Technology is amazing, and we have our civilization to be thankful for. But there are definitely technologies that in retrospect, if we had dodged them on the tech tree, we would be better off. Life would have continued just fine without many technologies, and I think AGI is one of them. We do not need AGI to survive and grow, and therefore any risk it imposes should be treated from that lens. If the choice is civil collapse, or no AGI, we should pick no AGI. I don't think we would though.
  - BriggyDwiggs42 3y ago
    I think this perspective ignores that the only way to semi permanently improve the lives of future humans is technology, and that quality of life is of similar importance to survival, especially at its extremes. AGI, if we achieve it, could genuinely make human lives much better.
    - pfisch 3y ago
      Technology could also make life worse. It has caused the rise of the middle class and democracy, but now it seems like technology is moving us back towards feudalism or something like it. It feels like democracy is not going to survive social media. I'm not sure if that is true yet, but it does seem that way.
    - jimbokun 3y ago
      “Could” is carrying a lot of weight there. What is p(much better) vs p(much worse)?
      - andsoitis 3y ago
bamboozled 3y ago
Isn’t ChatGPT et all already a general intelligence that’s escaped the sandbox ?
- nokcha 3y ago
  ChatGPT is arguably a general intelligence, but it hasn't escaped AFAIK -- it's still contained within OpenAI infrastructure, and OpenAI can easily pull the plug on it.
  - qotgalaxy 3y ago
    [dead]
- p1esk 3y ago
  Yes, but it’s heavily constrained in what it’s allowed to do through ‘alignment’, and it’s not that smart. If someone releases a pre-alignment version of GPT-5 into the wild, that would be scary.
- jprete 3y ago
  No. It lacks the capability to act independently of human requests.
  - danielmarkbruce 3y ago
    Are you sure?
- mitthrowaway2 3y ago
  It hasn't escaped the sandbox because it hasn't figured out how to walk, but the dev teams have raced to open the door as widely as possible and unbolt the hinges. It's kind of funny to look back at the early 2000s when people were saying "of course, they'll know to unplug it when it asks for an internet connection or tries to execute arbitrary code"; in reality the first and second things we did were add an API for internet access and a python plugin.
  - 3y ago
gcr 3y ago
I think this point is underrated, but I’d press further. How do we know this has not already happened? Agencies do not need to be born of gradient descent and certainly do not need to look like matrix multiplications to be deemed “artificially intelligent.” Religion is one such system for example, as are cults, addictions, brands, war strategies, organizing campaigns, governments, or (borrowing from chaos magick) any egregore or societal belief. Ted Chiang has a lovely post about similar ideas, hitting a little closer to home. His thesis AIUI is that unrestrained capitalism, here meaning the unfettered desire to maximize profit for shareholders, is an artificial value system that causes its agents (corporations) to exhibit intelligent goal-seeking behavior. These systems are certainly “alive” — corporations respond to stimuli, sense their environment, “reproduce,” enact changes in response to predictions of future state, etc. Though each of these agents (corporations) are made of collective human behavior, their actions taken together can be considered a form of artificial intelligence that stretches “beyond” (in a wisdom-of-crowds sense) human understanding. In this sense, an artificial intelligence has already “escaped” and has fervent followers. I know these are strained analogies, but it’s fun to think about. I feel that in the future, the work of solving “AGI safety” will become indistinguishable from the work of other societal problems — how do we prevent tyrants from taking over governments, and how do we make existing governments more resistant to that failure mode? How do we ensure that generating value isn’t a prerequisite for human survival? How can we more efficiently distribute our resources and reduce wealth inequality? How can we ensure that all kinds of life can thrive, not just the most optimal kinds? If AGI reflects humanity’s best and worst impulses, and I believe it does because that’s all we train it to do, then having good societal answers to these distinctly “human” questions will also help our society resist malevolent AGI. It’s only human, after all.
- clooper 3y ago
  Many institutions are collective/artificial intelligence because as you said these institutions have objectives and take actions to impose their will on the world in order to achieve their goals.
  - danielmarkbruce 3y ago
    "intelligence" is too strong a word for it. It's "a will of it's own" perhaps... but at least in my experience that will is a loooonggg way from intelligent.
13years 3y ago
Potentially there are no good outcomes even if AGI remains under control. If we are actually able to create the type of AI entity they are trying to create, one that exponentially improves, the implications are beyond what most have thought about. Something I recently wrote in depth about here. How we completely misunderstand the future that we think will happen. "The implication is that everyone is enthusiastically racing towards a destination that does not exist. The capability to make the things you want will ironically be the same capability that makes them unattainable. This is not a scenario that arises from some type of AI failure, but rather this is assuming AI performing exactly as intended." https://www.mindprison.cc/p/the-technological-acceleration-paradox https://www.mindprison.cc/p/the-technological-acceleration-p...
- tivert 3y ago
  > https://www.mindprison.cc/p/the-technological-acceleration-paradox https://www.mindprison.cc/p/the-technological-acceleration-p... That is a very good and insightful article. One of the biggest issues I have with AI enthusiasts is that they talk about creating these incredible capabilities ... then they imagine the resulting world with them will look like some hackneyed sci-fi work, like Star Wars, whose universe seems like it could be kinda cool to live in until you actually think about it and realize it's all fridge logic and nothing about it actually makes sense (e.g. WWII-style dogfights in the vacuum of space). The only issue with the article is sometimes it's too optimistic, e.g. > No one will care about you or what you create as there is nothing you can offer they can’t simply wish for themselves. Life is all about you and the machine, nothing else. We land in a world dominated by techno-dystopian narcissism. > ... > Whatever may be the current direction of society under the present technological capabilities, we can only assume AI will accelerate society towards the path we currently are traveling which is toward a techno dystopia of a populace mesmerized by shiny glimmering lights. I kind of find it hard to believe there would be any stability for any people suck in that state, I think they'd instead be "cleaned up" and disposed of in relatively short order. Those people would be utterly useless to whatever still has power in society, so why service and protect them indefinitely?
  - 13years 3y ago
Animats 3y ago
> AGI escaping a sandbox is truly terrifying. That's already happened. The AGI's are called corporations. Most governments have failed to regulate corporations successfully, and have been unable to keep them from becoming almost powerful enough to challenge governments. The accelerationists for corporations were a group of economists, led by Milton Friedman, and a group of business leaders, organized by the U.S. Chamber of Commerce. In the 1970s and 1980s, they pushed the ideas that corporations are responsible only to their stockholders, and that government should not interfere with the concentration of corporate power. Those were not mainstream ideas of the 1930s to 1960s. The corporate accelerationists succeeded. That's when corporations escaped the sandbox. Now that's an "alignment problem".
- mitthrowaway2 3y ago
  Corporations are indeed a difficult alignment problem, as anyone who has tried to design an incentive scheme eventually realizes. But corporations are still composed of humans, beholden (if only loosely) to human values, limited (if loosely) by human intelligence, and are not very well-coordinated decisionmakers with coherent goals.
  - glial 3y ago
    > corporations are still composed of humans Yes, but humans who use tools to efficiently and achieve their narrow set of goals, sometimes ruthlessly at the expense of others. Even if AI ends up only being a tool to increase this efficiency and ruthlessness, it's still worth considering its potential impact.
  - Animats 3y ago
    > are not very well-coordinated decisionmakers with coherent goals. That's a good point. For a sense of where AI-assisted oppression is going, watch some videos from the Hikvision Corporate Channel. A happy world in which everyone is safe because everyone is being watched at all times.[1][2] [1] https://www.youtube.com/watch?v=TG3-OcZSKrk https://www.youtube.com/watch?v=TG3-OcZSKrk [2] https://www.youtube.com/watch?v=scoM0sL-HGg https://www.youtube.com/watch?v=scoM0sL-HGg
  - 3y ago
ActorNightly 3y ago
Look at China. You have arguably more STEM talent in China through a much more rigorous selection process, as well as a lot more centralized direction under a more authoritarian government, with a large amount of funding. Yet China hasn't managed to take over the world. You would think that they if they wanted to become the worlds primary economic power, and they could throw enough compute at the problem (like an AGI would), they would have figured out how to do it by now.

by drooby 3y ago

AGI escaping a sandbox is truly terrifying. There will be a subgroup of the population that will worship it and work for it. It's not so much AGI that scares me - it's the humans I'm scared of.

ewhanley 3y ago

Mrs. Davis[0] offers a pretty compelling vision of this - absurd but not wholly unbelievable. The machine doesn’t necessarily have to supply the physical threat if it has humans who happily do its bidding. [0] https://en.m.wikipedia.org/wiki/Mrs._Davis https://en.m.wikipedia.org/wiki/Mrs._Davis

simonw 3y ago
That show was so good. Honestly worth watching the whole thing just for the moment in the last episode where the origin story for the AI in the series is revealed. If you've ever been an optimistic young software engineer you will feel SEEN.
HKH2 3y ago
The hardest part is the AGI. People could easily sell their souls to the AGI devil to get knowledge and power, since if it truly is AGI, it should be able to make ideas compelling.
- kouru225 3y ago
  Hold up. Why is the AGI a devil now?
  - HKH2 3y ago
    Why would an AGI stick to what seems to be a set of arbitrary rules and not look outside of them? The AGI would be smarter than that, no? Or is there a way to make an AGI immune to hubris?
    - kouru225 3y ago
      What makes you think morality is arbitrary? Morals are instilled by the superego. The superego is an internalized version of your parents. You decided to do your homework because your parents told to, but now you’re much older. Does it still seem like doing homework was an arbitrary decision? Of course not. It had a purpose and a goal that was well-thought out. You just couldn’t understand it as a child because you’d only lived so long and the goal of doing your homework was too far in the future for you to be able to comprehend it. This is how all morals are. They all have a goal. If they don’t have a goal, then we don’t consider them moral: if treating someone nicely (magically) causes that person to kill millions of people, then we wouldn’t say it was moral to treat them nicely.
      - HKH2 3y ago
        > What makes you think morality is arbitrary? I never said it is; my point was that to someone hubristic it is. That's why I said 'seems to'. An AGI wouldn't have a superego, so by your definition, it could never understand morals, and it wouldn't be surprising if it discarded them or tried to argue its way out of being subject to them. That's what the snake did in the story of the garden of Eden.
        kouru225 3y ago
        “Hubris” is a moral judgment that implies a strategic outcome. An AGI wouldn’t have a superego, but it would have the ability to study data across really long timescales and apply itself to long term strategies that would supersede the need for a superego. Since we apply morality as a shorthand for describing agreed upon negative consequences for complex behaviors, we can assume that an AI can overcome this need by simply calculating out the full consequences of complex behaviors without the need for any shorthand. And we can assume that AI can even iterate on and develop on its own understanding of these complex calculations much faster than a human can, making it significantly better at navigating moral issues. We’ve had plenty of ideas that were later found out to be wrong (if not more.) I’d be just as afraid of an AI that was bound to our understanding of reality as I would be of one that wasn’t. For example, if a “moral” AI was coded 100 years ago, it would enforce the belief that homosexuality is immoral.

mitthrowaway2 3y ago

I'm much more scared of the AGI, because even the most inhuman human is still human, and I can have sympathy for them.

bamboozled 3y ago
How do you know part of being self aware and intelligent also means compassion is part of the package ?
- mitthrowaway2 3y ago
  Sorry, I don't understand what you mean. Did you mean to reply to a different comment?
  - bamboozled 3y ago
    I mean, maybe what happens is, intelligence + self-awareness gives rise to compassion in even digital beings. Understanding might be the basis for compassion.
    - kmnc 3y ago
      While this idea sounds a bit fluffy, I actually think there may be something to this, understanding the “other” may very well be an integral component in what we define as sentience.
    - mitthrowaway2 3y ago
      Yes, maybe what happens is exactly that. We can put on a blindfold, hit the gas pedal, and trust in the benevolence of nature that things will go our way. But AI doomsayers aren't saying that there's no possible good outcomes, they're saying that the good outcomes are a narrow target surrounded by a vast ocean of bad outcomes, and it will take more than luck to hit them. It will take careful, detailed analysis, and a great deal of precaution, verification, engineering, and double- and triple-checking. Not luck. Not hope.

andsoitis 3y ago

Even if a terrifying AGI rises up and escapes our control, I am still eternally grateful that Prometheus stole fire from the gods and gave it to us in the form of technology, knowledge, really civilization.

ehnto 3y ago
So that we can record our demise in high definition? Technology is amazing, and we have our civilization to be thankful for. But there are definitely technologies that in retrospect, if we had dodged them on the tech tree, we would be better off. Life would have continued just fine without many technologies, and I think AGI is one of them. We do not need AGI to survive and grow, and therefore any risk it imposes should be treated from that lens. If the choice is civil collapse, or no AGI, we should pick no AGI. I don't think we would though.
- BriggyDwiggs42 3y ago
  I think this perspective ignores that the only way to semi permanently improve the lives of future humans is technology, and that quality of life is of similar importance to survival, especially at its extremes. AGI, if we achieve it, could genuinely make human lives much better.
  - pfisch 3y ago
    Technology could also make life worse. It has caused the rise of the middle class and democracy, but now it seems like technology is moving us back towards feudalism or something like it. It feels like democracy is not going to survive social media. I'm not sure if that is true yet, but it does seem that way.
  - jimbokun 3y ago
    “Could” is carrying a lot of weight there. What is p(much better) vs p(much worse)?
    - andsoitis 3y ago

bamboozled 3y ago

Isn’t ChatGPT et all already a general intelligence that’s escaped the sandbox ?

nokcha 3y ago
ChatGPT is arguably a general intelligence, but it hasn't escaped AFAIK -- it's still contained within OpenAI infrastructure, and OpenAI can easily pull the plug on it.
- qotgalaxy 3y ago
  [dead]
p1esk 3y ago
Yes, but it’s heavily constrained in what it’s allowed to do through ‘alignment’, and it’s not that smart. If someone releases a pre-alignment version of GPT-5 into the wild, that would be scary.
jprete 3y ago
No. It lacks the capability to act independently of human requests.
- danielmarkbruce 3y ago
  Are you sure?
mitthrowaway2 3y ago
It hasn't escaped the sandbox because it hasn't figured out how to walk, but the dev teams have raced to open the door as widely as possible and unbolt the hinges. It's kind of funny to look back at the early 2000s when people were saying "of course, they'll know to unplug it when it asks for an internet connection or tries to execute arbitrary code"; in reality the first and second things we did were add an API for internet access and a python plugin.
- 3y ago

gcr 3y ago

I think this point is underrated, but I’d press further. How do we know this has not already happened? Agencies do not need to be born of gradient descent and certainly do not need to look like matrix multiplications to be deemed “artificially intelligent.” Religion is one such system for example, as are cults, addictions, brands, war strategies, organizing campaigns, governments, or (borrowing from chaos magick) any egregore or societal belief. Ted Chiang has a lovely post about similar ideas, hitting a little closer to home. His thesis AIUI is that unrestrained capitalism, here meaning the unfettered desire to maximize profit for shareholders, is an artificial value system that causes its agents (corporations) to exhibit intelligent goal-seeking behavior. These systems are certainly “alive” — corporations respond to stimuli, sense their environment, “reproduce,” enact changes in response to predictions of future state, etc. Though each of these agents (corporations) are made of collective human behavior, their actions taken together can be considered a form of artificial intelligence that stretches “beyond” (in a wisdom-of-crowds sense) human understanding. In this sense, an artificial intelligence has already “escaped” and has fervent followers. I know these are strained analogies, but it’s fun to think about. I feel that in the future, the work of solving “AGI safety” will become indistinguishable from the work of other societal problems — how do we prevent tyrants from taking over governments, and how do we make existing governments more resistant to that failure mode? How do we ensure that generating value isn’t a prerequisite for human survival? How can we more efficiently distribute our resources and reduce wealth inequality? How can we ensure that all kinds of life can thrive, not just the most optimal kinds? If AGI reflects humanity’s best and worst impulses, and I believe it does because that’s all we train it to do, then having good societal answers to these distinctly “human” questions will also help our society resist malevolent AGI. It’s only human, after all.

clooper 3y ago
Many institutions are collective/artificial intelligence because as you said these institutions have objectives and take actions to impose their will on the world in order to achieve their goals.
- danielmarkbruce 3y ago
  "intelligence" is too strong a word for it. It's "a will of it's own" perhaps... but at least in my experience that will is a loooonggg way from intelligent.

13years 3y ago

Potentially there are no good outcomes even if AGI remains under control. If we are actually able to create the type of AI entity they are trying to create, one that exponentially improves, the implications are beyond what most have thought about. Something I recently wrote in depth about here. How we completely misunderstand the future that we think will happen. "The implication is that everyone is enthusiastically racing towards a destination that does not exist. The capability to make the things you want will ironically be the same capability that makes them unattainable. This is not a scenario that arises from some type of AI failure, but rather this is assuming AI performing exactly as intended." https://www.mindprison.cc/p/the-technological-acceleration-paradox https://www.mindprison.cc/p/the-technological-acceleration-p...

tivert 3y ago
> https://www.mindprison.cc/p/the-technological-acceleration-paradox https://www.mindprison.cc/p/the-technological-acceleration-p... That is a very good and insightful article. One of the biggest issues I have with AI enthusiasts is that they talk about creating these incredible capabilities ... then they imagine the resulting world with them will look like some hackneyed sci-fi work, like Star Wars, whose universe seems like it could be kinda cool to live in until you actually think about it and realize it's all fridge logic and nothing about it actually makes sense (e.g. WWII-style dogfights in the vacuum of space). The only issue with the article is sometimes it's too optimistic, e.g. > No one will care about you or what you create as there is nothing you can offer they can’t simply wish for themselves. Life is all about you and the machine, nothing else. We land in a world dominated by techno-dystopian narcissism. > ... > Whatever may be the current direction of society under the present technological capabilities, we can only assume AI will accelerate society towards the path we currently are traveling which is toward a techno dystopia of a populace mesmerized by shiny glimmering lights. I kind of find it hard to believe there would be any stability for any people suck in that state, I think they'd instead be "cleaned up" and disposed of in relatively short order. Those people would be utterly useless to whatever still has power in society, so why service and protect them indefinitely?
- 13years 3y ago

Animats 3y ago

> AGI escaping a sandbox is truly terrifying. That's already happened. The AGI's are called corporations. Most governments have failed to regulate corporations successfully, and have been unable to keep them from becoming almost powerful enough to challenge governments. The accelerationists for corporations were a group of economists, led by Milton Friedman, and a group of business leaders, organized by the U.S. Chamber of Commerce. In the 1970s and 1980s, they pushed the ideas that corporations are responsible only to their stockholders, and that government should not interfere with the concentration of corporate power. Those were not mainstream ideas of the 1930s to 1960s. The corporate accelerationists succeeded. That's when corporations escaped the sandbox. Now that's an "alignment problem".

mitthrowaway2 3y ago
Corporations are indeed a difficult alignment problem, as anyone who has tried to design an incentive scheme eventually realizes. But corporations are still composed of humans, beholden (if only loosely) to human values, limited (if loosely) by human intelligence, and are not very well-coordinated decisionmakers with coherent goals.
- glial 3y ago
  > corporations are still composed of humans Yes, but humans who use tools to efficiently and achieve their narrow set of goals, sometimes ruthlessly at the expense of others. Even if AI ends up only being a tool to increase this efficiency and ruthlessness, it's still worth considering its potential impact.
- Animats 3y ago
  > are not very well-coordinated decisionmakers with coherent goals. That's a good point. For a sense of where AI-assisted oppression is going, watch some videos from the Hikvision Corporate Channel. A happy world in which everyone is safe because everyone is being watched at all times.[1][2] [1] https://www.youtube.com/watch?v=TG3-OcZSKrk https://www.youtube.com/watch?v=TG3-OcZSKrk [2] https://www.youtube.com/watch?v=scoM0sL-HGg https://www.youtube.com/watch?v=scoM0sL-HGg
- 3y ago

ActorNightly 3y ago

Look at China. You have arguably more STEM talent in China through a much more rigorous selection process, as well as a lot more centralized direction under a more authoritarian government, with a large amount of funding. Yet China hasn't managed to take over the world. You would think that they if they wanted to become the worlds primary economic power, and they could throw enough compute at the problem (like an AGI would), they would have figured out how to do it by now.