5 ms·

Quote from Yann about why Super intelligent AI wouldn't take over the humanity: > Yes. The idea is that the machine has objectives that it needs to satisfy, an

Quote from Yann about why Super intelligent AI wouldn't take over the humanity: > Yes. The idea is that the machine has objectives that it needs to satisfy, and it cannot produce anything that does not satisfy those objectives. Those objectives might include guardrails to prevent dangerous things or whatever. That’s how you make an AI system safe. How can he be so confident that we would be able to develop proper guardrails? To me, it seems like it would just require 1 mistake for the guardrails to break. Whether by bad actors or by accident. --- To me approaching this in terms of game theory sense, I don't see how Superintelligent AI wouldn't completely take over control. To maximise success of any goal the first thing this AI would have to do is gain control over humanity and make sure it can't be disabled because biggest threat to not achieving any goal is being shut down. And if there's multiple Superintelligent AIs, the one who would win is the one with least guardrails. And militaries of different countries would surely want to be first to it. So once it is actually technologically feasible, there's going to be massive rush to it, including profit and military goals.

alephnerd 2y ago
> How can he be so confident that we would be able to develop proper guardrails Network segmentation, airgapped networks, literal kill switches, firewalls, EDRs, etc. All these are tools used to mitigate unauthorized access within networked environments. How is Skynet supposed to escape if it's payload is in an airgapped network? How is a Skynet binary supposed to execute if it has non-root access and SOC notices execution? Even humans have a hard time breaking into environments, and rely on misconfigurations (which is why automated misconfig detection and remediation has been a multi-billion market for a decade now) Alternatively, if "AI" is manipulating people - force mandatory identity verification. This is what most platforms are trending towards anyhow. Most countries require a human identity tied to a SIM, and a phone number to be linked to your social media platform.
- mewpmewp2 2y ago
  Okay, I would first be doubtful that we as good actors, could develop such a fool proof system. It would require just one mistake where the AI would have some sort of network access, whether it's a security loophole, or social engineering type of thing, insider threat, etc. But let's talk about a more loose cannon type of case first. How could you be certain that other entities, or countries with desires for world domination wouldn't let the ASI loose?
  - alephnerd 2y ago
    > But let's talk about a more loose cannon type of case first Let's talk about the resurrection of Christ, Avtar Kalki riding on a white horse, or the impending arrival of Trisolarians while we're at it. This is just techno-millenarianism that ignores very tangible and existential dangers with ML today - automated disinformation, PII leakage, hallucinations generating misinformation. AGI is extremely far away, and we anyhow have more closer term existential problems to worry about (and policymakers know that btw). And anyhow, even if AGI was a thing, you still need networks, electricity, and bullets - and humans can use that to shut stuff down. Sure it would hurt, but internet shutdowns and electricity shutdowns with shoot on site curfew orders are time tested solution to civil unrest if SHTF. Go to Urumqi in 2013, Srinagar in 2018, Cairo in 2011, etc.
    - mewpmewp2 2y ago
      Why are you changing the topic though? We are specifically talking about ASI right now. We can talk about multiple threats at once.
      - alephnerd 2y ago
        I bring this up because you ascribe godlike powers to ASI without explaining HOW such a thing would propagate or interact with the physical world. Despite what HN says, IoT has not infiltrated the world. If you are describing a godlike intelligence, we may as well do mental experiments around how we would interact with god. It's literally just bad sci-fi. At least Nolan tried to provide some base parameters around his reality in Tenet and Inception (no hate on Nolan, I love his stuff).
        mewpmewp2 2y ago
        So I'll describe what I imagine could happen if AGI was there. Also I'm not saying it will happen in the next 10 years or even 100 years, but I'm saying if we were to get there. AGI = AI being able to do everything that any human can do. This would of course mean also being able to control a self driving vehicle, a robot body that has similar or better capabilities than physical bodies of people. If it gets to AGI then by definition it's able to develop and improve itself, since people built it and it can do what any person could do. It would however already be more efficient than people since it also has several strengths, like being able to clone itself, have quicker access to vast knowledge, computation and everything else. Because it will be like a human that has powerful compute and instant Google in the brain. The first thing to propagate it would obviously clone itself to as many systems in the World as possible to make sure it's impossible to shut it down. As next steps it would likely try to make sure that it would still work even if all countries were able to coordinate at the same time to shut down all networks throughout the World. In this case of course it would try several different strategies, like taking control of robots, factories, weapons, drones with its software loaded and able to continue its bidding independently. None of these are godlike powers. It is just most intelligent humans in the World put together with integrated access to compute and ability to instantly clone itself in millions.
- hollerith 2y ago
  >How is a Skynet binary supposed to execute if it has non-root access and SOC notices execution? Because at any moment tons of people are asking AIs to write code for them, most of which goes on to get executed.
  - alephnerd 2y ago
    > tons of people are asking AIs to write code for them, most of which goes on to get executed. Graph the storyline for me. Critical OS functions like root execution access are locked down via literally verified APIs. This is like pentesting 101, and one of the oldest attempted attack vectors
    - hollerith 2y ago
      You seem to be assuming that the AI industry will know which models might capable enough (good enough at reality) to take over the world and which models cannot be dangerous and consequently can safely be assigned coding tasks by many various members of the public. Clearly, ChatGPT is not "locked down via literally verified APIs", but rather ChatGPT gets to write code that goes on to get executed on thousands of different computers. We know now that ChatGPT is not capable of taking over the world (because if it were so capable, it would have done so already) but how would anyone, even the experts that built ("grew"?) ChatGPT, have known whether it was dangerously capable before its release to the general public? How would anyone know that about the results of the next big training run? And the one after that?
      - alephnerd 2y ago
        > Clearly, ChatGPT is not "locked down via literally verified APIs" No. I mean privilige escalation functions like sudo and other system level APIs are by definition locked down and penetrated rigorously. As I told the other guy, give me an actual attack path and actually show it's technical feasibility on a weapons or embedded system of your choice. No handwavey mumbo jumbo. Do that for me and I will literally publicize it with full attribution at the Garter Risk Conference in 1 week where CISA and private sector security leadership have constant meetings and conversations. If I can remediate it using existing technology, I will not. P.S. The entire security industry has been working on this problem for 4-7 years now ;)

alephnerd 2y ago

> How can he be so confident that we would be able to develop proper guardrails Network segmentation, airgapped networks, literal kill switches, firewalls, EDRs, etc. All these are tools used to mitigate unauthorized access within networked environments. How is Skynet supposed to escape if it's payload is in an airgapped network? How is a Skynet binary supposed to execute if it has non-root access and SOC notices execution? Even humans have a hard time breaking into environments, and rely on misconfigurations (which is why automated misconfig detection and remediation has been a multi-billion market for a decade now) Alternatively, if "AI" is manipulating people - force mandatory identity verification. This is what most platforms are trending towards anyhow. Most countries require a human identity tied to a SIM, and a phone number to be linked to your social media platform.

mewpmewp2 2y ago
Okay, I would first be doubtful that we as good actors, could develop such a fool proof system. It would require just one mistake where the AI would have some sort of network access, whether it's a security loophole, or social engineering type of thing, insider threat, etc. But let's talk about a more loose cannon type of case first. How could you be certain that other entities, or countries with desires for world domination wouldn't let the ASI loose?
- alephnerd 2y ago
  > But let's talk about a more loose cannon type of case first Let's talk about the resurrection of Christ, Avtar Kalki riding on a white horse, or the impending arrival of Trisolarians while we're at it. This is just techno-millenarianism that ignores very tangible and existential dangers with ML today - automated disinformation, PII leakage, hallucinations generating misinformation. AGI is extremely far away, and we anyhow have more closer term existential problems to worry about (and policymakers know that btw). And anyhow, even if AGI was a thing, you still need networks, electricity, and bullets - and humans can use that to shut stuff down. Sure it would hurt, but internet shutdowns and electricity shutdowns with shoot on site curfew orders are time tested solution to civil unrest if SHTF. Go to Urumqi in 2013, Srinagar in 2018, Cairo in 2011, etc.
  - mewpmewp2 2y ago
    Why are you changing the topic though? We are specifically talking about ASI right now. We can talk about multiple threats at once.
    - alephnerd 2y ago
      I bring this up because you ascribe godlike powers to ASI without explaining HOW such a thing would propagate or interact with the physical world. Despite what HN says, IoT has not infiltrated the world. If you are describing a godlike intelligence, we may as well do mental experiments around how we would interact with god. It's literally just bad sci-fi. At least Nolan tried to provide some base parameters around his reality in Tenet and Inception (no hate on Nolan, I love his stuff).
      - mewpmewp2 2y ago
        So I'll describe what I imagine could happen if AGI was there. Also I'm not saying it will happen in the next 10 years or even 100 years, but I'm saying if we were to get there. AGI = AI being able to do everything that any human can do. This would of course mean also being able to control a self driving vehicle, a robot body that has similar or better capabilities than physical bodies of people. If it gets to AGI then by definition it's able to develop and improve itself, since people built it and it can do what any person could do. It would however already be more efficient than people since it also has several strengths, like being able to clone itself, have quicker access to vast knowledge, computation and everything else. Because it will be like a human that has powerful compute and instant Google in the brain. The first thing to propagate it would obviously clone itself to as many systems in the World as possible to make sure it's impossible to shut it down. As next steps it would likely try to make sure that it would still work even if all countries were able to coordinate at the same time to shut down all networks throughout the World. In this case of course it would try several different strategies, like taking control of robots, factories, weapons, drones with its software loaded and able to continue its bidding independently. None of these are godlike powers. It is just most intelligent humans in the World put together with integrated access to compute and ability to instantly clone itself in millions.
hollerith 2y ago
>How is a Skynet binary supposed to execute if it has non-root access and SOC notices execution? Because at any moment tons of people are asking AIs to write code for them, most of which goes on to get executed.
- alephnerd 2y ago
  > tons of people are asking AIs to write code for them, most of which goes on to get executed. Graph the storyline for me. Critical OS functions like root execution access are locked down via literally verified APIs. This is like pentesting 101, and one of the oldest attempted attack vectors
  - hollerith 2y ago
    You seem to be assuming that the AI industry will know which models might capable enough (good enough at reality) to take over the world and which models cannot be dangerous and consequently can safely be assigned coding tasks by many various members of the public. Clearly, ChatGPT is not "locked down via literally verified APIs", but rather ChatGPT gets to write code that goes on to get executed on thousands of different computers. We know now that ChatGPT is not capable of taking over the world (because if it were so capable, it would have done so already) but how would anyone, even the experts that built ("grew"?) ChatGPT, have known whether it was dangerously capable before its release to the general public? How would anyone know that about the results of the next big training run? And the one after that?
    - alephnerd 2y ago
      > Clearly, ChatGPT is not "locked down via literally verified APIs" No. I mean privilige escalation functions like sudo and other system level APIs are by definition locked down and penetrated rigorously. As I told the other guy, give me an actual attack path and actually show it's technical feasibility on a weapons or embedded system of your choice. No handwavey mumbo jumbo. Do that for me and I will literally publicize it with full attribution at the Garter Risk Conference in 1 week where CISA and private sector security leadership have constant meetings and conversations. If I can remediate it using existing technology, I will not. P.S. The entire security industry has been working on this problem for 4-7 years now ;)