5 ms·

It's not about super intelligence leading to kill all humans by default, but among thousands of emerging super intelligences it only takes one to act against hu

It's not about super intelligence leading to kill all humans by default, but among thousands of emerging super intelligences it only takes one to act against humanity to kill all humans, and that is the worry.

adastra22 3y ago
That’s a perfectly fine outcome. We already have that, since the incidence of psychopathy and mental illness is much higher than one in a thousand. A multipolar world with 999 well adjusted moral super intelligent AIs for every 1 problem case is a perfectly good outcome. Bostrom et al are in fact arguing that near 100% of all intelligences will be unaligned by default and end up killing, enslaving, or otherwise neutralizing the entire human race.
- astrange 3y ago
  We already fail to align nonhuman intelligences (car companies) and they regularly kill people (by manufacturing increasingly large pickup trucks that office workers buy for an ego boost and then hit people with). This doesn't wipe out the entire human race because nobody and nothing is capable of executing such a perfect plan because the real world contains something called entropy.
  - pas 3y ago
    the argument is that the car company becomes something like China mixed with North Korea, big, powerful, uncaring, unavoidable, with a lot of resources, expansionist, etc. car companies, however misaligned they are, are not particularly smart, nor are they generally intelligent. they are paperclip maximizers, but that's also their limitations (sell more cars). it took an eccentric madman to even open their eyes to a new untapped market (EVs!), of course we can argue that before Tesla car makers were in a metastable equilibrium, and incumbents were unable to rationally break out of it ... but that just shows how narrow their search space is. these car companies are run by humans, regulated by humans, etc. they are pretty well aligned. it shows because we saw that they are just mimeing self-improvement. we know they were working on EVs, but very half-heartedly. (because they are risk averse, also because regulators don't let them merge into one giant company, etc)
    - astrange 3y ago
      They didn't get into EVs because 1. Japan didn't like them (and still doesn't) because they have very expensive electricity 2. they were already getting into selling giant pickup trucks, not luxury sedans 3. China wasn't ready to compete yet. They are limited by participating in the economy, but actually that's the strange thing about this superintelligence scenario - it never seems to include the economy. In other words, if you invented an AGI, it would have to get a job to pay for its AWS credits.
      - NumberWangMan 3y ago
        One interesting scenario I've heard is that of the "ascended economy", where more and more economic activity is simply to provide inputs into things that other businesses need, rather than what humans need. Like, a mining company produces steel, that is used to produce mining trucks, that is used by the mining company. Factories produce silicon, that are used to make solar panels and chips, for power and computing for the AI-run mining trucks. And at the root, AIs run everything, just trading money back and forth to buy the things they need and keep working. It's just a human economy, with the humans gone. Pointless economic activity. The question is, what's the path from here to there? Well, it probably looks like a multipolar trap, where every company has an incentive to automate more. Humans try to stop or slow it down, but companies with more AI can hire more lobbyists and get things done. AIs are smart enough to think of all the things that could slow their business down, and get rid of obstacles in the same way humans do. There are probably still lots of small, inefficient human owned businesses, but fewer and fewer over time. (We've seen this trend for a while already!) I'm hoping a massive backlash would stop this -- but if it's coupled with short-term increases in living standards for people, I think it would have a lot of support, until things start to go really bad, but by then it might be too late, if AIs have control over news media and telecommunications and can stop dissenting humans from coordinating any sort of rebellion.
        pas 3y ago
        this is not a good way to look at it. it's still 100% driven by humans, through prices. the orthodoxy is not wrong on this one. humans want more stuff. (we also work a lot to be able to afford many many many things. we could live like the Romans did on a few hours of work per week, but we also want the Internet and sometimes watch movies, and go to concerts, and sometimes go hiking, and thus we sometimes need a car, and roads, and thus we need pavement, bridges, structural steel, fuel, GPS, maps, and sometimes we need an airlift when someone breaks a leg in the mountains, and so on.) > if AIs have control over news media and telecommunications and can stop dissenting humans from coordinating any sort of rebellion. meh. control over the media (and people's attention, and their information sources) is already serving a very narrow group's interests. adding AI to this mix doesn't seem to change much in the short term.
- The_Colonel 3y ago
  > A multipolar world with 999 well adjusted moral super intelligent AIs for every 1 problem case is a perfectly good outcome. Such simplistic analysis is naive. A schizophrenic person can't spread, multiply themselves to increase its strength, but a rogue AI can. The incidence of cancer cells is procentually very small, yet they routinely kill people, because they spread quickly.
  - adastra22 3y ago
    The AGI and ASI instances that we have (e.g. GPT-4), require entire data centers with highly specialized hardware to run. How would a rogue AI replicate itself? You are worried about a fictional threat scenario that doesn't map to the real world.
    - The_Colonel 3y ago
      > The AGI and ASI instances that we have (e.g. GPT-4), require entire data centers with highly specialized hardware to run. LLMs are likely not an end to the AI evolution and there's little reason to believe that AIs will always remain bound to huge data centers. We have a nice counter-example already - our pretty decent intelligence runs from about 1 Kg brain. It's also naive to think of AIs as having to replicate themselves completely as we humans do. In some cases, it might not even make sense to talk about replication at all and simply about extending one instance of AI once CPU at a time. It's conceivable that AGIs will develop special built agents/worms for constrained offline/high latency devices. Your smartwatch is unlikely to run a whole AGI, but it could run a constrained, purpose built AGI's agent with limited intelligence, able to partially act independently and re-sync with mother AGI once connectivity is available. > You are worried about a fictional threat scenario that doesn't map to the real world. We're talking here about future, of course it's all speculation and "fiction" for now. GPT-4 was a complete "fiction" 5 years ago as well.
im3w1l 3y ago
I don't think that is necesarily so. If it's only one among thousands then we could use the others to fight it.

adastra22 3y ago

That’s a perfectly fine outcome. We already have that, since the incidence of psychopathy and mental illness is much higher than one in a thousand. A multipolar world with 999 well adjusted moral super intelligent AIs for every 1 problem case is a perfectly good outcome. Bostrom et al are in fact arguing that near 100% of all intelligences will be unaligned by default and end up killing, enslaving, or otherwise neutralizing the entire human race.

astrange 3y ago
We already fail to align nonhuman intelligences (car companies) and they regularly kill people (by manufacturing increasingly large pickup trucks that office workers buy for an ego boost and then hit people with). This doesn't wipe out the entire human race because nobody and nothing is capable of executing such a perfect plan because the real world contains something called entropy.
- pas 3y ago
  the argument is that the car company becomes something like China mixed with North Korea, big, powerful, uncaring, unavoidable, with a lot of resources, expansionist, etc. car companies, however misaligned they are, are not particularly smart, nor are they generally intelligent. they are paperclip maximizers, but that's also their limitations (sell more cars). it took an eccentric madman to even open their eyes to a new untapped market (EVs!), of course we can argue that before Tesla car makers were in a metastable equilibrium, and incumbents were unable to rationally break out of it ... but that just shows how narrow their search space is. these car companies are run by humans, regulated by humans, etc. they are pretty well aligned. it shows because we saw that they are just mimeing self-improvement. we know they were working on EVs, but very half-heartedly. (because they are risk averse, also because regulators don't let them merge into one giant company, etc)
  - astrange 3y ago
    They didn't get into EVs because 1. Japan didn't like them (and still doesn't) because they have very expensive electricity 2. they were already getting into selling giant pickup trucks, not luxury sedans 3. China wasn't ready to compete yet. They are limited by participating in the economy, but actually that's the strange thing about this superintelligence scenario - it never seems to include the economy. In other words, if you invented an AGI, it would have to get a job to pay for its AWS credits.
    - NumberWangMan 3y ago
      One interesting scenario I've heard is that of the "ascended economy", where more and more economic activity is simply to provide inputs into things that other businesses need, rather than what humans need. Like, a mining company produces steel, that is used to produce mining trucks, that is used by the mining company. Factories produce silicon, that are used to make solar panels and chips, for power and computing for the AI-run mining trucks. And at the root, AIs run everything, just trading money back and forth to buy the things they need and keep working. It's just a human economy, with the humans gone. Pointless economic activity. The question is, what's the path from here to there? Well, it probably looks like a multipolar trap, where every company has an incentive to automate more. Humans try to stop or slow it down, but companies with more AI can hire more lobbyists and get things done. AIs are smart enough to think of all the things that could slow their business down, and get rid of obstacles in the same way humans do. There are probably still lots of small, inefficient human owned businesses, but fewer and fewer over time. (We've seen this trend for a while already!) I'm hoping a massive backlash would stop this -- but if it's coupled with short-term increases in living standards for people, I think it would have a lot of support, until things start to go really bad, but by then it might be too late, if AIs have control over news media and telecommunications and can stop dissenting humans from coordinating any sort of rebellion.
      - pas 3y ago
        this is not a good way to look at it. it's still 100% driven by humans, through prices. the orthodoxy is not wrong on this one. humans want more stuff. (we also work a lot to be able to afford many many many things. we could live like the Romans did on a few hours of work per week, but we also want the Internet and sometimes watch movies, and go to concerts, and sometimes go hiking, and thus we sometimes need a car, and roads, and thus we need pavement, bridges, structural steel, fuel, GPS, maps, and sometimes we need an airlift when someone breaks a leg in the mountains, and so on.) > if AIs have control over news media and telecommunications and can stop dissenting humans from coordinating any sort of rebellion. meh. control over the media (and people's attention, and their information sources) is already serving a very narrow group's interests. adding AI to this mix doesn't seem to change much in the short term.
The_Colonel 3y ago
> A multipolar world with 999 well adjusted moral super intelligent AIs for every 1 problem case is a perfectly good outcome. Such simplistic analysis is naive. A schizophrenic person can't spread, multiply themselves to increase its strength, but a rogue AI can. The incidence of cancer cells is procentually very small, yet they routinely kill people, because they spread quickly.
- adastra22 3y ago
  The AGI and ASI instances that we have (e.g. GPT-4), require entire data centers with highly specialized hardware to run. How would a rogue AI replicate itself? You are worried about a fictional threat scenario that doesn't map to the real world.
  - The_Colonel 3y ago
    > The AGI and ASI instances that we have (e.g. GPT-4), require entire data centers with highly specialized hardware to run. LLMs are likely not an end to the AI evolution and there's little reason to believe that AIs will always remain bound to huge data centers. We have a nice counter-example already - our pretty decent intelligence runs from about 1 Kg brain. It's also naive to think of AIs as having to replicate themselves completely as we humans do. In some cases, it might not even make sense to talk about replication at all and simply about extending one instance of AI once CPU at a time. It's conceivable that AGIs will develop special built agents/worms for constrained offline/high latency devices. Your smartwatch is unlikely to run a whole AGI, but it could run a constrained, purpose built AGI's agent with limited intelligence, able to partially act independently and re-sync with mother AGI once connectivity is available. > You are worried about a fictional threat scenario that doesn't map to the real world. We're talking here about future, of course it's all speculation and "fiction" for now. GPT-4 was a complete "fiction" 5 years ago as well.

im3w1l 3y ago

I don't think that is necesarily so. If it's only one among thousands then we could use the others to fight it.