6 ms·

The meme that AGI, if we ever have it, will somehow endanger humanity is just stupid to me. For one, the previous US president is the perfect illustration that

The meme that AGI, if we ever have it, will somehow endanger humanity is just stupid to me. For one, the previous US president is the perfect illustration that intelligence is neither sufficient nor necessary for gaining power in this world. And we do in fact live in a world where the upper echelons of power mostly interact in the decidedly analog spaces of leadership summits, high-end restaurants, golf courses and country clubs. Most world leaders interact with a real computer like a handful of times per year. Furthermore, due to the warring nature of us humans, the important systems in the world like banking, electricity, industrial controls, military power etc. are either air-gapped or have a requirement for multiple humans to push physical buttons in order to actually accomplish scary things. And because we humans are a bit stupid and make mistakes sometimes, like fat-fingering an order on the stock market and crashing everything, we have completely manual systems that undo mistakes and restore previous values. Sure, a mischievous AGI could do some annoying things. But nothing that our human enemies existing today couldn't also do. The AGI won't be able to guess encryption keys any faster than the dumb old computer it runs on. Simply put, to me there is no plausible mechanism by which the supposedly extremely intelligent machine would assert its dominance over humanity. We have plenty of scary-smart humans in the world and they don't go around becoming super-villains either.

Tepix 4y ago
> The meme that AGI, if we ever have it, will somehow endanger humanity is just stupid to me. we have never encountered an entity with superhuman intelligence. Clearly it is hard to predict what is going to happen. There is an unknown risk.
HL33tibCe7 4y ago
It sounds like you haven’t really thought through AI safety in any real detail at all. Airgapping and the necessity of human input are absolutely not ways to prevent an AGI gaining access to a system. A true, superintelligent AGI could easily extort (or persuade) those humans. If you think concerns over AGI are “stupid”, you haven’t thought about it enough. It’s a massive display of ignorance. The Computerphile AI safety videos are an approachable introduction to this topic. Edit: just as one very simple example, can you even imagine the destruction that could (probably will) occur if (when) a superintelligent AGI gets access to the internet? Imagine the zero days it could discover and exploit, for whatever purpose it felt necessary. And this is just the tip of the iceberg, just one example off the top of my head of something that would almost inevitably be a complete catastrophe.
- semi-extrinsic 4y ago
  So what if the AGI discovers a bunch of zero days on the internet? We can just turn the entire internet off for a week and be just fine, remember? And exactly how does the AGI extort or persuade humans? What can it say to me that you can't say to me right now?
  - luma 4y ago
    Sends you a text from your spouse's phone number that an emergency has happened and you need to go to xyz location right now. Someone else is a gun owner and they get a similar text, but their spouse is being held captive and are sent to the same location with a description of you as the kidnapper. Scale this as desired. Use your imagination!
    - semi-extrinsic 4y ago
      Again, what part of this couldn't an evil human group already do today? What in this plan requires an AGI for it to be a scary scenario? Of course if you do it to everyone, total gridlock ensues and nobody gets to the place where they would be killed. If you only do it to a few, maybe there will be a handful of killings before everyone learns not to trust a text ever again. As an aside, I agree living in a country where people will take their guns around and try to solve things vigilante style instead of calling the police, is a very bad thing in general. In all first-world countries except one, and in almost all developing countries in the world, this is a solved problem.
      - luma 4y ago
        Scale! Do this to two people, no problem you can get a team of folks to dig through their social media, come up with a plan, execute. Might take a day to get a good crisis cooked up. An AI might be able to run this attack against the entire planet, all at once. Try to think like a not-human, and give yourself the capacity of near-infinite scale. What could you do? Human systems are hilariously easy to disrupt and you don't need nukes to make that happen.
- 4y ago
jdmoreira 4y ago
Humans can't iterate themselves over generations in short periods of time. An AGI is only bound by whatever computing power it has access to. And if it's smart it can gain access to a lot of computing power (think about a computer worm spreading itself on the internet)
sabellito 4y ago
There's a wondeful youtube channel from a researcher who focusses exactly on this topic, I think you should check it out: https://www.youtube.com/watch?v=ZeecOKBus3Q https://www.youtube.com/watch?v=ZeecOKBus3Q
- semi-extrinsic 4y ago
  I watched the whole thing. Man spent a lot of breath asserting that an AGI will have broadly the same types of goals that humans do. Said exactly zero words about why we won't just be able to tell the AGI "no, you're not getting what you want", and then turn it off.
  - dinosaurdynasty 4y ago
    He covers that in another video: https://www.youtube.com/watch?v=3TYT1QfdfsM https://www.youtube.com/watch?v=3TYT1QfdfsM
    - semi-extrinsic 4y ago
      Oh FFS. He starts off by giving the AGI a robot body that can kill things (don't do that in the first place!), and then he adds a stop button in a type of location that wouldn't pass muster on a table saw, let alone an industrial robot. If you've placed the stop button such that you cannot access it safely in all possible failure modes of the dangerous machine, be it bench grinder or an AGI or whatever, you have failed the very basics of industrial safety and you must go home and be sad, for you will not be receiving a cookie. And the thought experiments of this guy are completely incoherent. At the same time the robot has human-level (or greater) intelligence, yet is extremely stupid, and you can somehow program it in an extremely simplistic way. Sorry, but all you've convinced me of is that this is just a random guy on YouTube with no particular education of relevance nor other qualifications (I checked), who is very pleased about his own ideas and is quite lacking in the ability to think critically.
      - sabellito 4y ago
viraptor 4y ago
> Furthermore, due to the warring nature of us humans, the important systems in the world like banking, electricity, industrial controls, military power etc. are either air-gapped or have a requirement for multiple humans to push physical buttons in order to actually accomplish scary things. Well, I have bad news for you. Airgap is very rarely a thing and even when it is people do stupid things. Examples from each extreme: all the industrial control systems remote desktops you can find with shodan on one side and Stuxnet on the other. > Sure, a mischievous AGI could do some annoying things. But nothing that our human enemies existing today couldn't also do. Think Wargames. You don't need to do something. You just need to lie to people in power in a convincing way.
blibble 4y ago
none of these things are air-gapped once you have the ability to coerce people if you want a fictional example: watch Colossus: The Forbin Project
- donkarma 4y ago
  i couldn't convince a cat to get into a cage
- semi-extrinsic 4y ago
  How does the AGI get this magical ability to coerce people? We couldn't even get half the population to wear a face mask after bombaring them with coercion for weeks on end.
  - blibble 4y ago
    watch the film
    - semi-extrinsic 4y ago
      Sorry, I just wasted ten minutes watching a YouTube video that someone else in this thread asserted was going to explain everything. Said video was entirely tangential to the point in question - why can't we just tell the AGI to sod off with it's wishes and goals. At the end of the day it is all just bits going through wires, which we are perfectly capable of disconnecting. Now I'm not going to spend a couple of hours watching a film that has a mediocre IMDB rating. But I do notice that the synopsis says "they gave the AGI total control over the US nuclear weapons arsenal". Yeah, let's not do that, I agree fully on this point. If we are concerned a thing will be evil and/or coercive, let's not give it weapons or other means by which to enact coercion. This is kind of my point - the AGI won't just stumble over the nuclear codes in a Reddit thread. For the AGI to actually accomplish something of real-world relevance, us humans have to agree.
      - blibble 4y ago
        > Now I'm not going to spend a couple of hours watching a film that has a mediocre IMDB rating. your loss > For the AGI to actually accomplish something of real-world relevance, us humans have to agree. Donald Trump managed to get control of the US nuclear arsenal

Tepix 4y ago

> The meme that AGI, if we ever have it, will somehow endanger humanity is just stupid to me. we have never encountered an entity with superhuman intelligence. Clearly it is hard to predict what is going to happen. There is an unknown risk.

HL33tibCe7 4y ago

It sounds like you haven’t really thought through AI safety in any real detail at all. Airgapping and the necessity of human input are absolutely not ways to prevent an AGI gaining access to a system. A true, superintelligent AGI could easily extort (or persuade) those humans. If you think concerns over AGI are “stupid”, you haven’t thought about it enough. It’s a massive display of ignorance. The Computerphile AI safety videos are an approachable introduction to this topic. Edit: just as one very simple example, can you even imagine the destruction that could (probably will) occur if (when) a superintelligent AGI gets access to the internet? Imagine the zero days it could discover and exploit, for whatever purpose it felt necessary. And this is just the tip of the iceberg, just one example off the top of my head of something that would almost inevitably be a complete catastrophe.

semi-extrinsic 4y ago
So what if the AGI discovers a bunch of zero days on the internet? We can just turn the entire internet off for a week and be just fine, remember? And exactly how does the AGI extort or persuade humans? What can it say to me that you can't say to me right now?
- luma 4y ago
  Sends you a text from your spouse's phone number that an emergency has happened and you need to go to xyz location right now. Someone else is a gun owner and they get a similar text, but their spouse is being held captive and are sent to the same location with a description of you as the kidnapper. Scale this as desired. Use your imagination!
  - semi-extrinsic 4y ago
    Again, what part of this couldn't an evil human group already do today? What in this plan requires an AGI for it to be a scary scenario? Of course if you do it to everyone, total gridlock ensues and nobody gets to the place where they would be killed. If you only do it to a few, maybe there will be a handful of killings before everyone learns not to trust a text ever again. As an aside, I agree living in a country where people will take their guns around and try to solve things vigilante style instead of calling the police, is a very bad thing in general. In all first-world countries except one, and in almost all developing countries in the world, this is a solved problem.
    - luma 4y ago
      Scale! Do this to two people, no problem you can get a team of folks to dig through their social media, come up with a plan, execute. Might take a day to get a good crisis cooked up. An AI might be able to run this attack against the entire planet, all at once. Try to think like a not-human, and give yourself the capacity of near-infinite scale. What could you do? Human systems are hilariously easy to disrupt and you don't need nukes to make that happen.
4y ago

jdmoreira 4y ago

Humans can't iterate themselves over generations in short periods of time. An AGI is only bound by whatever computing power it has access to. And if it's smart it can gain access to a lot of computing power (think about a computer worm spreading itself on the internet)

sabellito 4y ago

There's a wondeful youtube channel from a researcher who focusses exactly on this topic, I think you should check it out: https://www.youtube.com/watch?v=ZeecOKBus3Q https://www.youtube.com/watch?v=ZeecOKBus3Q

semi-extrinsic 4y ago
I watched the whole thing. Man spent a lot of breath asserting that an AGI will have broadly the same types of goals that humans do. Said exactly zero words about why we won't just be able to tell the AGI "no, you're not getting what you want", and then turn it off.
- dinosaurdynasty 4y ago
  He covers that in another video: https://www.youtube.com/watch?v=3TYT1QfdfsM https://www.youtube.com/watch?v=3TYT1QfdfsM
  - semi-extrinsic 4y ago
    Oh FFS. He starts off by giving the AGI a robot body that can kill things (don't do that in the first place!), and then he adds a stop button in a type of location that wouldn't pass muster on a table saw, let alone an industrial robot. If you've placed the stop button such that you cannot access it safely in all possible failure modes of the dangerous machine, be it bench grinder or an AGI or whatever, you have failed the very basics of industrial safety and you must go home and be sad, for you will not be receiving a cookie. And the thought experiments of this guy are completely incoherent. At the same time the robot has human-level (or greater) intelligence, yet is extremely stupid, and you can somehow program it in an extremely simplistic way. Sorry, but all you've convinced me of is that this is just a random guy on YouTube with no particular education of relevance nor other qualifications (I checked), who is very pleased about his own ideas and is quite lacking in the ability to think critically.
    - sabellito 4y ago

viraptor 4y ago

> Furthermore, due to the warring nature of us humans, the important systems in the world like banking, electricity, industrial controls, military power etc. are either air-gapped or have a requirement for multiple humans to push physical buttons in order to actually accomplish scary things. Well, I have bad news for you. Airgap is very rarely a thing and even when it is people do stupid things. Examples from each extreme: all the industrial control systems remote desktops you can find with shodan on one side and Stuxnet on the other. > Sure, a mischievous AGI could do some annoying things. But nothing that our human enemies existing today couldn't also do. Think Wargames. You don't need to do something. You just need to lie to people in power in a convincing way.

blibble 4y ago

none of these things are air-gapped once you have the ability to coerce people if you want a fictional example: watch Colossus: The Forbin Project

donkarma 4y ago
i couldn't convince a cat to get into a cage
semi-extrinsic 4y ago
How does the AGI get this magical ability to coerce people? We couldn't even get half the population to wear a face mask after bombaring them with coercion for weeks on end.
- blibble 4y ago
  watch the film
  - semi-extrinsic 4y ago
    Sorry, I just wasted ten minutes watching a YouTube video that someone else in this thread asserted was going to explain everything. Said video was entirely tangential to the point in question - why can't we just tell the AGI to sod off with it's wishes and goals. At the end of the day it is all just bits going through wires, which we are perfectly capable of disconnecting. Now I'm not going to spend a couple of hours watching a film that has a mediocre IMDB rating. But I do notice that the synopsis says "they gave the AGI total control over the US nuclear weapons arsenal". Yeah, let's not do that, I agree fully on this point. If we are concerned a thing will be evil and/or coercive, let's not give it weapons or other means by which to enact coercion. This is kind of my point - the AGI won't just stumble over the nuclear codes in a Reddit thread. For the AGI to actually accomplish something of real-world relevance, us humans have to agree.
    - blibble 4y ago
      > Now I'm not going to spend a couple of hours watching a film that has a mediocre IMDB rating. your loss > For the AGI to actually accomplish something of real-world relevance, us humans have to agree. Donald Trump managed to get control of the US nuclear arsenal