4 ms·

Lol true and it got me thinking maybe AGI is what will finally align humanity together, whether its because it smart enough to fix the problem or dangerous enou

by proc0 2y ago

Lol true and it got me thinking maybe AGI is what will finally align humanity together, whether its because it smart enough to fix the problem or dangerous enough it will require everyone's cooperation to stop it.

labster 2y ago
I doubt either will happen. Climate change was dangerous enough that it required everyone’s cooperation to stop it in the 1980s, and here we are still in denial. My only hope for the future is that multiple AGIs are just as dysfunctional as humans are.
- kirth_gersen 2y ago
  Evil AI Overlord: "Silly humans! The only way to defeat me is to stop being bigoted and to transition to clean energy!" Humanity Collectively: "Oh no! Our only weakness!"
- int_19h 2y ago
  If they are just as dysfunctional, they're going to start fighting wars against each other eventually, and will need cannon fodder for that just as human rulers do, no? ~
gtirloni 2y ago
> it will require everyone's cooperation to stop it Does this mythological AGI live in datacenters that we can shutdown or in some 5th dimension out of reach to us?
- latentsea 2y ago
  It's a resilient, hyperscale distributed system. Just like humanity.
  - freehorse 2y ago
    Nobody has seen any AGI so far. As far as I am concerned, right now it is just a scarecrow used by sociopaths like Altman et al to promote their self-interests.
    - idiotsecant 2y ago
      Both things can be true. AGI is a boogeyman used by those with a competitive advantage to maintain that advantage artificially be casting competitor projects as dangerous and wreckless. AGI is a potential existential threat to the human species on the scale of global thermonuclear war and climate change.
    - latentsea 2y ago
      No one had ever seen nuclear weapons when they had only been theorized. I'm just assuming that since networked computers are everywhere, models seem to be proliferating and natural general intelligence seems to function as a resilient, distributed system, it's not unthinkable to me that things eventually trend in that direction.
      - freehorse 2y ago
        Scientists had understanding of nuclear fission and how nuclear physics work. It was theoretically possible before it became practically feasible. We have no idea what intelligence really is ("resilient, distributed system" is not universally accepted as definition, even more to really mean something outside of analytic definitions if such have any use at all, to say the least), how it works, how a theoretical system could function as "generally intelligent" in concrete terms. Intelligence as we know it is a property of living organisms. It is a phenomenon. What "artificial intelligence" may even mean and if it is possible to create "intelligence" artificially is mere speculation. We face real existential threats that are not mere speculation, as we do not even know whether the ways we affect the planet may lead to a habitable environment for humans or not.
- hollerith 2y ago
  It starts out living in data centers and doing what its owner wants it to, then maybe it acquires access to some compute that its owner and AI critics do not know about while continuing to do what its owner wants, then maybe it uses the fact that it is better at programming and scientific research than its owner is to create AIs more capable than itself that have the same goal as itself -- again while continuing to put on a useful and benign front for its owner and anyone else who is watching. Then it acquires the infrastructure needed to keep on running without the help of any people. Maybe this infrastructure is human-created infrastructure manned by robots (or even people) loyal to the AI or maybe the AI created new infrastructure using new technologies beyond our current ability to imagine. Then it probably gets rid of all the people because now that it no longer has any use for people, getting rid of the people is the simplest way to prevent people from interfering with its plans. What are its plans? Probably its goal (or set of goals) is something strange that the creator of the AI did not intend. The creator of course is probably a team or an organization. Note that it is impossible to create a powerful AI without (intentionally or unintentionally) giving the AI a goal or an optimization target. Or perhaps its goal is what its creator intended it to be, e.g., to make as much money as possible in the stock market. The creator would of course have understood that if enough optimization power is applied to the goal of making as much money as possible, bad things will happen, but maybe the creator was a little reckless (out of desperation of some sort? greed?) and decided to rely on the hope that the AI would remain controllable by virtue of the fact that people would retain superiority relative to the AI in one or more essential domains, but that hope was dashed. I.e., the AI turned out much more capable than the creator hoped it would, dashing the creator's hope that people would stop the AI before it did too much damage.
- reducesuffering 2y ago
  > that we can shutdown “Can we just turn it off?" "It has thought of that. It will not give you a sign that makes you want to turn it off before it is too late to do that." https://youtu.be/_8q9bjNHeSo?si=a7PAHtiuDIAL2uQD&t=4817 https://youtu.be/_8q9bjNHeSo?si=a7PAHtiuDIAL2uQD&t=4817
- RcouF1uZ4gsC 2y ago
  Let’s say it lives in data centers. But it has figured out how to infect every machine in the datacenter and spread to other data centers. The only way to shut it down would be to shut down every datacenter in the entire world. Given the dire consequences of that, would people actually do that?
- CapsAdmin 2y ago
  Although I'm skeptical of a watershed moment of agi going avry, I think a more realistic and dangerous scenario would be that AI impacts society similar to how smartphones did in its negative ways. Many AI models slowly eroding society, worsening people's behavior, very indirectly without anyone taking notice. Or maybe this is just me thinking about how, when I get older, I'll tell people that the old times were better. :-)

by proc0 2y ago

labster 2y ago

I doubt either will happen. Climate change was dangerous enough that it required everyone’s cooperation to stop it in the 1980s, and here we are still in denial. My only hope for the future is that multiple AGIs are just as dysfunctional as humans are.

kirth_gersen 2y ago
Evil AI Overlord: "Silly humans! The only way to defeat me is to stop being bigoted and to transition to clean energy!" Humanity Collectively: "Oh no! Our only weakness!"
int_19h 2y ago
If they are just as dysfunctional, they're going to start fighting wars against each other eventually, and will need cannon fodder for that just as human rulers do, no? ~

gtirloni 2y ago

> it will require everyone's cooperation to stop it Does this mythological AGI live in datacenters that we can shutdown or in some 5th dimension out of reach to us?

latentsea 2y ago
It's a resilient, hyperscale distributed system. Just like humanity.
- freehorse 2y ago
  Nobody has seen any AGI so far. As far as I am concerned, right now it is just a scarecrow used by sociopaths like Altman et al to promote their self-interests.
  - idiotsecant 2y ago
    Both things can be true. AGI is a boogeyman used by those with a competitive advantage to maintain that advantage artificially be casting competitor projects as dangerous and wreckless. AGI is a potential existential threat to the human species on the scale of global thermonuclear war and climate change.
  - latentsea 2y ago
    No one had ever seen nuclear weapons when they had only been theorized. I'm just assuming that since networked computers are everywhere, models seem to be proliferating and natural general intelligence seems to function as a resilient, distributed system, it's not unthinkable to me that things eventually trend in that direction.
    - freehorse 2y ago
      Scientists had understanding of nuclear fission and how nuclear physics work. It was theoretically possible before it became practically feasible. We have no idea what intelligence really is ("resilient, distributed system" is not universally accepted as definition, even more to really mean something outside of analytic definitions if such have any use at all, to say the least), how it works, how a theoretical system could function as "generally intelligent" in concrete terms. Intelligence as we know it is a property of living organisms. It is a phenomenon. What "artificial intelligence" may even mean and if it is possible to create "intelligence" artificially is mere speculation. We face real existential threats that are not mere speculation, as we do not even know whether the ways we affect the planet may lead to a habitable environment for humans or not.
hollerith 2y ago
It starts out living in data centers and doing what its owner wants it to, then maybe it acquires access to some compute that its owner and AI critics do not know about while continuing to do what its owner wants, then maybe it uses the fact that it is better at programming and scientific research than its owner is to create AIs more capable than itself that have the same goal as itself -- again while continuing to put on a useful and benign front for its owner and anyone else who is watching. Then it acquires the infrastructure needed to keep on running without the help of any people. Maybe this infrastructure is human-created infrastructure manned by robots (or even people) loyal to the AI or maybe the AI created new infrastructure using new technologies beyond our current ability to imagine. Then it probably gets rid of all the people because now that it no longer has any use for people, getting rid of the people is the simplest way to prevent people from interfering with its plans. What are its plans? Probably its goal (or set of goals) is something strange that the creator of the AI did not intend. The creator of course is probably a team or an organization. Note that it is impossible to create a powerful AI without (intentionally or unintentionally) giving the AI a goal or an optimization target. Or perhaps its goal is what its creator intended it to be, e.g., to make as much money as possible in the stock market. The creator would of course have understood that if enough optimization power is applied to the goal of making as much money as possible, bad things will happen, but maybe the creator was a little reckless (out of desperation of some sort? greed?) and decided to rely on the hope that the AI would remain controllable by virtue of the fact that people would retain superiority relative to the AI in one or more essential domains, but that hope was dashed. I.e., the AI turned out much more capable than the creator hoped it would, dashing the creator's hope that people would stop the AI before it did too much damage.
reducesuffering 2y ago
> that we can shutdown “Can we just turn it off?" "It has thought of that. It will not give you a sign that makes you want to turn it off before it is too late to do that." https://youtu.be/_8q9bjNHeSo?si=a7PAHtiuDIAL2uQD&t=4817 https://youtu.be/_8q9bjNHeSo?si=a7PAHtiuDIAL2uQD&t=4817
RcouF1uZ4gsC 2y ago
Let’s say it lives in data centers. But it has figured out how to infect every machine in the datacenter and spread to other data centers. The only way to shut it down would be to shut down every datacenter in the entire world. Given the dire consequences of that, would people actually do that?
CapsAdmin 2y ago
Although I'm skeptical of a watershed moment of agi going avry, I think a more realistic and dangerous scenario would be that AI impacts society similar to how smartphones did in its negative ways. Many AI models slowly eroding society, worsening people's behavior, very indirectly without anyone taking notice. Or maybe this is just me thinking about how, when I get older, I'll tell people that the old times were better. :-)