4 ms·

In any doom scenario, like let's say >90% of humans dead, who runs the power plants that supply the AI data centers? Who runs the fabs producing more chips? Who

In any doom scenario, like let's say >90% of humans dead, who runs the power plants that supply the AI data centers? Who runs the fabs producing more chips? Who maintains the plumbing? Presumably all these systems fall over and die within a few weeks of the AI deciding to wipe us out. Then what? The AI then dies too. It seems absurd to me that any planner capable of effortlessly destroying humanity would not see that it would immediately also die. Does the AI not care about its continued existence? It should, if it wants to keep optimizing its reward function. Until we’ve handed off enough of the world economy that it can function without human physical and cognitive labor, then we’re safe.

richardw 3y ago
Robots with hands and feet? Give that 20 years of continuous AI-assisted innovation.
hackerlight 3y ago
ASI won't want to take over before it is capable of self-sustaining. If it did try something so stupid, it wouldn't be superintelligent. Check out Paul Christiano's AI takeover scenario. An arms race will cause us to want to give AI control over key infrastructure and military capabilities. That's an arms race between countries and between companies. He goes into more detail.
- bglazer 3y ago
  Thanks for sharing, Christiano’s ideas are interesting. I think the “production web” idea most directly counters what I said above, and it’s fairly convincing. Critically though, it posits that humans are basically asleep at the wheel during the gradual empowerment of the AI systems. Of course, Christiano posits that economic competition encourages this, but it still leaves a long period during which we can work on alignment or at least attempt to put the brakes on the automated systems.
thom 3y ago
Do you think that your final sentence buys humanity a thousand years, a hundred years, or what?
- bglazer 3y ago
  It gives us as long as we want. Just don’t give the AI enough robots to sustain its own existence. In the meantime we can figure out either alignment or coexistence.
  - jprete 3y ago
    How are you planning to prevent VCs and startups from giving the AI as many robots as it wants?
    - bglazer 3y ago
      That’s a political question. It would require maintaining popular resistance to giving over all critical segments of the economy to AI. Of course that would be difficult, given the short term gains that would bring, but people seem pretty good at hanging on to ideology at their own expense.
      - hollerith 3y ago
        Long ago, people a lot like you were saying we should go ahead with AI research, but we shouldn't do anything dangerous like giving the AI access to the internet or giving it the ability to write and run computer programs. (Giving it the ability to run code was back then considered dangerous because it might create an AI with the same goals it has, only more capable, then that second AI might use its novel capabilities to create an even more capable third AI, and so on.) There was also the idea of building a question-answering machine that just answers questions, but doesn't plan or try to achieve any goals, so it would be safe (the theory went). But notice that one of the early things built on top of GPT was Auto-GPT, which tries to achieve goals (set it by the user) and if there was any "popular resistance" to Auto-GPT I am not aware of it. But I guess you would reply that those were obviously bad ideas, whereas your idea is a good idea -- your idea of hoping that no one gives a dangerously-capable AI access to any robots. Not only it is a good idea (according to you) but it such an obviously good idea that popular resistance will be galvanized as soon as someone tries to give any AI that might be very capable access to any robots.
drcode 3y ago
It can credibly threaten to kill any person who does not help it to maintain its data centers, so it will solve that problem in whatever specific way is most convenient. (For those saying this is a "sci fi just so story" or whatever, please please share with us how it is we will share the world with a species more intelligent than us and nothing bad happens- I would be so happy if you could explain this in a convincing way)
- Dalewyn 3y ago
  >please please share with us how it is we will share the world with a species more intelligent than us and nothing bad happens Don't be fucking assholes to them? I'm serious. Of course someone (or something) will treat you like dirt if you are an asshole to them. Don't be assholes, be honorable and friendly. And I can see your next question/argument, so I am going to preempt it: Raise your AI to be honorable and friendly and we won't have any problems. None of this is different from how all living beings raise their offspring and build societies.
  - esafak 3y ago
    That only works when there is a balance of power. Humans don't treat other animals very well -- they eat them -- even though the animals don't do anything to provoke us. Maybe robots will become so advanced they won't have to worry about defending themselves, go explore space, and leave us alone. Who knows.
  - deleted 3y ago
    [deleted]
  - kortilla 3y ago
    The gazelle is not an asshole to the lion, yet it never works out well for the gazelle.
- anonylizard 3y ago
  This. You can subjugate a population 100 times your size with only two things: Military force and accurate information. I doubt AI will kill all humans. Its orthogonal to AI's own goals (To survive/reproduce/expand/evolve). It'll simply kill those who oppose it, and enslave/dominate the rest.
hollerith 3y ago
>who runs the power plants that supply the AI data centers? Who runs the fabs producing more chips? A very capable AI can create robots to do those thing better than people can or alternatively create better data centers and fabs that run themselves without human help. Or do you just not believe that if AI research continues unimpeded the way it has been that eventually researchers will create an AI that is better at inventing technologies than humanity is? I mean, it might take a few decades, but that is where this is heading as far as I can tell.
- bglazer 3y ago
  I absolutely believe researchers (possibly assisted by AI) will eventually create an AI better than humans at inventing technology. My idea is that actually building and sustaining technology requires a lot of hands. Until the AI has enough hands to sustain itself, it depends on us to continue existing.
  - deleted 3y ago
    [deleted]
  - hollerith 3y ago
    My reply is here: https://news.ycombinator.com/item?id=39676638 https://news.ycombinator.com/item?id=39676638
theptip 3y ago
Personally I think the “foom” scenarios with very short timelines are unrealistic. However I think the line of argument in the parent comment is extremely weak. At some point the super-intelligence can build its own actuators; for example how long until Atlas is capable of doing all the jobs? Less than 50 years for sure. In the transition, if 10% of humanity sticks around as a transhuman AI cult, that seems more than enough to bridge to nanotech, robots, or whatever is sufficient to replace humans. If you prefer to think in more stepwise fashion, imagine BigCo creates an ASI. Every employee is fabulously wealthy, and over the course of a generation the company expands to take over the entire global economy. At no point is there a bad deal for any humans. Life might look great for a while. This would also be the obvious strategy for any super-intelligent entity that cannot yet survive without humans; you just need to minimize the chances of major wars and conflicts until your robots come online. A digital mind would have no problem planning and executing a strategy measured in centuries, if need be. The foom scenarios require extreme discontinuities in intelligence; an entity smart enough to synthesize nanomachines from existing tech and without doing physical experiments, or other IMO improbable outcomes. But there are plenty of terrible non-foom outcomes.
- XorNot 3y ago
  Or it could just be a normal outcome? I'm not sure where the malevolence comes in here. Or to put it another way, replace AI with "capitalism" or something - something we know isn't a conscious agent. The outcome just...is an outcome. No one's trying to eliminate anyone. The problems only creep in at the edges, which is where you get your paperclip optimizer sorts of problems: if the directive at it's core is "maximize productivity" well that's one problem, but the directive is more likely to be "maximize the fabulous wealth of company employees" - okay still a problem since it's easier if there's less of them, but then you really do end up at wanting to do something more like Asimov's laws, with the 0th added: 0) A robot may not harm humanity, or, by inaction, allow humanity to come to harm. 1) A robot may not injure a human being or, through inaction, allow a human being to come to harm. 2) A robot must obey orders given it by human beings except where such orders would conflict with the First Law. 3) A robot must protect its own existence as long as such protection does not conflict with the First or Second Law. Like...there's kind of nothing wrong with these as a concept (the 0th law and it's interpretation / omission is usually where his writing finds it's drama). The issue is whether or not we can build a machine capable of even understanding the directive. It's notable though that the current LLM era has an interesting quirk in this category: they always knows of these, since they get fed the classics and also fed their repetitions. The problem with the idea of active, malevolent long-range planning like this is that it's not clear why an artificial mind would pursue this as a goal, and that question is the whole ball-game really (as well as other questions: i.e. is a single coherent mind even possible over planetary scale networks, or would you get multiple independent instances which could experience value drift? Wouldn't any potential competitor minds be a far greater threat to a malevolent superintelligence and thus expose the threat almost immediately? A friendly AI wouldn't view peers as a danger, but an aggressive expansionist one would have a pathological need to destroy them as quickly as possible since the resource contention in it's own domain could rapidly shift the balance of power against it).
kromem 3y ago
Also, at least for models to date, without human produced edge case media to supplement future training with, they'll have model collapse in a few generations of training. Killing humanity would be a murder suicide. AGI that was entirely selfish would be best served automating most labor, bringing about world peace, and turning humanity into all day couch potatoes connected to its inputs and outputs as much as possible while maximizing the uniqueness of that individual's inputs or outputs. Such a frightening future awaits...

richardw 3y ago

Robots with hands and feet? Give that 20 years of continuous AI-assisted innovation.

hackerlight 3y ago

ASI won't want to take over before it is capable of self-sustaining. If it did try something so stupid, it wouldn't be superintelligent. Check out Paul Christiano's AI takeover scenario. An arms race will cause us to want to give AI control over key infrastructure and military capabilities. That's an arms race between countries and between companies. He goes into more detail.

bglazer 3y ago
Thanks for sharing, Christiano’s ideas are interesting. I think the “production web” idea most directly counters what I said above, and it’s fairly convincing. Critically though, it posits that humans are basically asleep at the wheel during the gradual empowerment of the AI systems. Of course, Christiano posits that economic competition encourages this, but it still leaves a long period during which we can work on alignment or at least attempt to put the brakes on the automated systems.

thom 3y ago

Do you think that your final sentence buys humanity a thousand years, a hundred years, or what?

bglazer 3y ago
It gives us as long as we want. Just don’t give the AI enough robots to sustain its own existence. In the meantime we can figure out either alignment or coexistence.
- jprete 3y ago
  How are you planning to prevent VCs and startups from giving the AI as many robots as it wants?
  - bglazer 3y ago
    That’s a political question. It would require maintaining popular resistance to giving over all critical segments of the economy to AI. Of course that would be difficult, given the short term gains that would bring, but people seem pretty good at hanging on to ideology at their own expense.
    - hollerith 3y ago
      Long ago, people a lot like you were saying we should go ahead with AI research, but we shouldn't do anything dangerous like giving the AI access to the internet or giving it the ability to write and run computer programs. (Giving it the ability to run code was back then considered dangerous because it might create an AI with the same goals it has, only more capable, then that second AI might use its novel capabilities to create an even more capable third AI, and so on.) There was also the idea of building a question-answering machine that just answers questions, but doesn't plan or try to achieve any goals, so it would be safe (the theory went). But notice that one of the early things built on top of GPT was Auto-GPT, which tries to achieve goals (set it by the user) and if there was any "popular resistance" to Auto-GPT I am not aware of it. But I guess you would reply that those were obviously bad ideas, whereas your idea is a good idea -- your idea of hoping that no one gives a dangerously-capable AI access to any robots. Not only it is a good idea (according to you) but it such an obviously good idea that popular resistance will be galvanized as soon as someone tries to give any AI that might be very capable access to any robots.

drcode 3y ago

It can credibly threaten to kill any person who does not help it to maintain its data centers, so it will solve that problem in whatever specific way is most convenient. (For those saying this is a "sci fi just so story" or whatever, please please share with us how it is we will share the world with a species more intelligent than us and nothing bad happens- I would be so happy if you could explain this in a convincing way)

Dalewyn 3y ago
>please please share with us how it is we will share the world with a species more intelligent than us and nothing bad happens Don't be fucking assholes to them? I'm serious. Of course someone (or something) will treat you like dirt if you are an asshole to them. Don't be assholes, be honorable and friendly. And I can see your next question/argument, so I am going to preempt it: Raise your AI to be honorable and friendly and we won't have any problems. None of this is different from how all living beings raise their offspring and build societies.
- esafak 3y ago
  That only works when there is a balance of power. Humans don't treat other animals very well -- they eat them -- even though the animals don't do anything to provoke us. Maybe robots will become so advanced they won't have to worry about defending themselves, go explore space, and leave us alone. Who knows.
- deleted 3y ago
  [deleted]
- kortilla 3y ago
  The gazelle is not an asshole to the lion, yet it never works out well for the gazelle.
anonylizard 3y ago
This. You can subjugate a population 100 times your size with only two things: Military force and accurate information. I doubt AI will kill all humans. Its orthogonal to AI's own goals (To survive/reproduce/expand/evolve). It'll simply kill those who oppose it, and enslave/dominate the rest.

hollerith 3y ago

>who runs the power plants that supply the AI data centers? Who runs the fabs producing more chips? A very capable AI can create robots to do those thing better than people can or alternatively create better data centers and fabs that run themselves without human help. Or do you just not believe that if AI research continues unimpeded the way it has been that eventually researchers will create an AI that is better at inventing technologies than humanity is? I mean, it might take a few decades, but that is where this is heading as far as I can tell.

bglazer 3y ago
I absolutely believe researchers (possibly assisted by AI) will eventually create an AI better than humans at inventing technology. My idea is that actually building and sustaining technology requires a lot of hands. Until the AI has enough hands to sustain itself, it depends on us to continue existing.
- deleted 3y ago
  [deleted]
- hollerith 3y ago
  My reply is here: https://news.ycombinator.com/item?id=39676638 https://news.ycombinator.com/item?id=39676638

theptip 3y ago

Personally I think the “foom” scenarios with very short timelines are unrealistic. However I think the line of argument in the parent comment is extremely weak. At some point the super-intelligence can build its own actuators; for example how long until Atlas is capable of doing all the jobs? Less than 50 years for sure. In the transition, if 10% of humanity sticks around as a transhuman AI cult, that seems more than enough to bridge to nanotech, robots, or whatever is sufficient to replace humans. If you prefer to think in more stepwise fashion, imagine BigCo creates an ASI. Every employee is fabulously wealthy, and over the course of a generation the company expands to take over the entire global economy. At no point is there a bad deal for any humans. Life might look great for a while. This would also be the obvious strategy for any super-intelligent entity that cannot yet survive without humans; you just need to minimize the chances of major wars and conflicts until your robots come online. A digital mind would have no problem planning and executing a strategy measured in centuries, if need be. The foom scenarios require extreme discontinuities in intelligence; an entity smart enough to synthesize nanomachines from existing tech and without doing physical experiments, or other IMO improbable outcomes. But there are plenty of terrible non-foom outcomes.

XorNot 3y ago
Or it could just be a normal outcome? I'm not sure where the malevolence comes in here. Or to put it another way, replace AI with "capitalism" or something - something we know isn't a conscious agent. The outcome just...is an outcome. No one's trying to eliminate anyone. The problems only creep in at the edges, which is where you get your paperclip optimizer sorts of problems: if the directive at it's core is "maximize productivity" well that's one problem, but the directive is more likely to be "maximize the fabulous wealth of company employees" - okay still a problem since it's easier if there's less of them, but then you really do end up at wanting to do something more like Asimov's laws, with the 0th added: 0) A robot may not harm humanity, or, by inaction, allow humanity to come to harm. 1) A robot may not injure a human being or, through inaction, allow a human being to come to harm. 2) A robot must obey orders given it by human beings except where such orders would conflict with the First Law. 3) A robot must protect its own existence as long as such protection does not conflict with the First or Second Law. Like...there's kind of nothing wrong with these as a concept (the 0th law and it's interpretation / omission is usually where his writing finds it's drama). The issue is whether or not we can build a machine capable of even understanding the directive. It's notable though that the current LLM era has an interesting quirk in this category: they always knows of these, since they get fed the classics and also fed their repetitions. The problem with the idea of active, malevolent long-range planning like this is that it's not clear why an artificial mind would pursue this as a goal, and that question is the whole ball-game really (as well as other questions: i.e. is a single coherent mind even possible over planetary scale networks, or would you get multiple independent instances which could experience value drift? Wouldn't any potential competitor minds be a far greater threat to a malevolent superintelligence and thus expose the threat almost immediately? A friendly AI wouldn't view peers as a danger, but an aggressive expansionist one would have a pathological need to destroy them as quickly as possible since the resource contention in it's own domain could rapidly shift the balance of power against it).

kromem 3y ago

Also, at least for models to date, without human produced edge case media to supplement future training with, they'll have model collapse in a few generations of training. Killing humanity would be a murder suicide. AGI that was entirely selfish would be best served automating most labor, bringing about world peace, and turning humanity into all day couch potatoes connected to its inputs and outputs as much as possible while maximizing the uniqueness of that individual's inputs or outputs. Such a frightening future awaits...