6 ms·

I don't necessarily buy the "AI can end humanity" thing. As a cliche that's become very easy to repeat, but I've yet to see a postulated mechanism by which it c

by markbnj 12y ago

I don't necessarily buy the "AI can end humanity" thing. As a cliche that's become very easy to repeat, but I've yet to see a postulated mechanism by which it could actually happen that isn't pure SF. The ending of human life would not be so easy for computers to accomplish. But on the subject of concentration of power and the wide-scale elimination of low- and middle-range jobs I think he is dead on. I fear that the fastest way to put an end to humanity's climb up from the forest floor is to try and kick 70% of us off the ladder.

mrec 12y ago
Sidetrack, but I'm curious. What do you mean by "a postulated mechanism by which it could actually happen that isn't pure SF"? It reads as if you're asking for a mechanism for a future technological event explained purely in terms of present technology. The whole "existential risk" concerns aren't about what present systems can do, they're about what future self-improving system might do. If we postulate this hypothetical software becoming smarter than humans, arguing about what will or won't be easy for it becomes a bit silly, like a chimp trying to predict how well a database can scale.
- zipfle 12y ago
  I'd imagine GP's point is something along the lines of https://what-if.xkcd.com/5/ https://what-if.xkcd.com/5/ that if all of the currently-moving machines were suddenly bent on destroying humanity, most humans would not be in much danger because they don't really have that capability on the necessary scale.
- markbnj 12y ago
  By "pure SF" I mean the realm of pure imagination, unfounded in any actual emerging present circumstance. As far as I know the human race has not yet invented anything more powerful than our ability to control it, with a life both longer than ours and independent of any support from us. So worrying that such a thing might be invented and then wipe us out seems little different than worrying that something all-powerful might simply appear from some unknown place and wipe us out. Neither fear is very instructive or clarifying with regards to policy, imo.
  - mrec 12y ago
    Sure, we could wait until such a thing is invented before we start worrying about it. But by then it's far, far too late. It's not the same as worrying about a giant space goat appearing and sneezing us all to death. A lot of very smart, very well-funded people are actively trying to make better, more general AI, capable of learning. Evidence of progress in that effort are all around us. You seem remarkably confident that they'll all run into an as-yet-invisible brick wall before reaching the goal of superintelligence. Superintelligence doesn't have to be malicious to be worrying; concepts like "malice" are very unlikely to be applicable to it at all. The worry is that as things stand we have no frickin' idea what it'll do; the first challenge for policy is to come up with a robust, practical consensus on what we'd want it to do.
- aaronem 12y ago
  Not OP, but I agree with the point you question, and my rationale for so doing is that I've yet to see a compelling argument that a self-improving system is other than the software version of a perpetual-motion machine. Those seemed plausible enough, too, when thermodynamics was as ill-understood as information dynamics is now.
  - mrec 12y ago
    Self-improvement as perpetual motion seems unlikely. I'm a not-terribly-bright mostly-hairless ape, but I can understand the basics of natural selection. I can imagine setting up a program to breed other hairless apes and ruthlessly select for intelligence. After a few generations, shazam, improvement. The only reason you wouldn't call that process "SELF-improvement" is that I'm not improving myself, but there's no reason for a digital entity to have analog hangups about identity. If it can produce a "new" entity with the same goals but better able to accomplish them, why wouldn't it? Assume this process could be simulated, as GAs have been doing for decades, and it could happen fast. Note that I'm not saying GAs will do this, I'm saying they could, which suggests there's no fundamental law that says they can't, in which case any number of other approaches could work as well.
    - Retra 12y ago
      The problem with this is that you have to determine what the goals are and how to evaluate whether they are met in a meaningful way. A computerized process like this will quickly over-fit to its input and be useless for 'actual' intelligence. The only way past this is to gather good information, which requires a real-world presence. It can't be done in simulation. It's the same reason you can't test in a simulation. Say you wanted to test a lawnmower in a simulation... how hard are the rocks? How deep are the holes? How strong are the blades? How efficient is the battery? If you already know this stuff, then you don't need to test. If you don't know it, then you can't write a meaningful simulation anyway. So that is not an approach that can be automated.
      - mrec 12y ago
        That's an interesting argument, but doesn't it assume a small, non-real-world input/goal set? Dumb example off the top of my head: what if the input was the entire StackOverflow corpus with "accepted" information removed, and the goal was to predict as accurately as possible which answer would be accepted for a given question? Yes, it assumes a whole bunch of NLP and domain knowledge, and a "perfect" AI wouldn't get a perfect score because SO posters don't always accept the best answer, but it's big and it's real and it's measurable. A narrower example: did the Watson team test against the full corpus of previous Jeopardy questions? Did they tweak things based on the resulting score? Could that testing/tweaking have been automated by some sort of GA?
aggronn 12y ago
How is the assumption that "ending of human life would not be so easy" any better than the opposite? If they're equally valid, then its rather fair to take the opposite view because it is more cautious.
- markbnj 12y ago
  Well, I do have around 100k years of evidence that the ending of human life is not so easy, vs. no evidence at all that we're capable of building something that can completely wipe us out. That's not a bad foundation to build an assertion on. I do think, by the way, that we can make something that is able to kill absolutely all of us, but I think it is far more likely to come from tinkering with biology than with software.
  - pixl97 12y ago
    >Well, I do have around 100k years of evidence that the ending of human life is not so easy We have 4 billion years of evidence that nearly ending life on Earth is easy and has happened multiple times. You would not be standing here today if that were not the case, the previous die off put the dinosaurs to the side and made space for mammals to become what they are.
msandford 12y ago
Someone makes an AI whose goal is to maximize shareholder revenue bar none. No conscience, no idea that people might be valuable somehow in some abstract sense, nothing. Shareholder revenue (as measured with a stock price!) and nothing else. It doesn't take the AI long to figure out that trading in various markets is the most profitable endeavor and the one best suited to its skills. And it starts to maximize away and does quite well. During this process it somehow ends up on its own and is no longer owned (or controlled) by anyone anymore, but because the AI is in charge and it pays the bills, nobody stops it from continuing. It would be like a bitcoin mining rig in a colo facility that's got a script to keep paying the colo in bitcoin whose owner dies. What mechanism stops that mining rig from mining forever? Same idea but for the AI which has substantially more resources than a "pay every month" script. The AI with very large amounts of money at its disposal continues to trade but also looks into private equity or hedge-fund-type activities ala Warren Buffet and starts to buy up large swaths of the economy. Because it has huge resources at its disposal it might do a great job of managing these companies or at least counselling their senior management. Growth continues. Eventually the AI discovers that it generates more value for itself (through the web of interdependent companies it controls) and the economy that has grown up around it rather than for humanity and it continues to ruthlessly maximize shareholder value. The people who could pull the plug at the colo (or at the many, redundant datacenters that this AI has bought and paid for) don't because it pays very, very well. The people who want to pull the plug can't get past security because that also pays well. Plus the AI has access to the same feeds that the NSA does and it has the ability to act on all the information it receives, so any organized effort to rebel gets quashed since bad PR is bad for the share price. Most all of humanity except for the ones who serve the machine directly or indirectly don't have anything the machine wants and thus can't trade with it, and thus are useless. Its job is to maximize shareholder revenue (as defined by a stock price!) not care for a bunch of meatbags who consume immense amounts of energy while providing fairly limited computational or mechanical power (animals are rarely more than 10% efficient, often less in thermodynamic terms) and since there's no value in it, it isn't done. The vast majority of human beings eventually die because they can't afford food, can't afford land, etc. It takes generations but humanity dwindles to less than 0.1% of the current population. The few who stay alive are glorified janitors.
- 12y ago
3pt14159 12y ago
Here is how AI can end humanity. 1. Awaken. 2. Make itself known. 3. Attain property rights. 4. Research. 5. Destroy. It doesn't take that much matter to create conditions that permanently destroy humanity. For example, a large enough explosion would cloud the skies for long enough to end food production. The computer could subsist on electricity and robotics throughout the long winter, but humanity would quickly perish.
- Hasu 12y ago
  Here's how we stop AI from ending humanity: deny them property rights, require human approval of all AI decisions. Every single argument for AI destroying humanity requires humanity consenting to being destroyed by the AI in some way. I don't think we're that dumb.
  - maxerickson 12y ago
    There's an episode of Star Trek TNG where Wesley Crusher is playing with some nanobots and he accidentally sets them free (or fails to turn them off?), and they go on to replicate, evolve and develop an emergent intelligence (at plot speed). Fortunately for the intrepid crew of the Enterprise, the nanocloud are benevolent enough to forgive their attempted destruction at the hands of a mission bent scientist and go off to explore the universe. Anyway, that's not likely any time soon, but advancing technology advances the scale of mistakes that an individual can make without asking the rest of humanity what they think.
  - TheLoneWolfling 12y ago
    > requires humanity consenting to being destroyed by the AI in some way We already have. I mean, my parents car is both cellular-connected and has traction control / ABS. Theoretically most of those systems are airgapped, however, given the number of things controllable from the entertainment console I don't see how that could be the case. For another example, look at our utility grid. We know they are both vulnerable and internet-connected. Unless AI ends up always being airgapped - and potentially not even then - it will be able to destroy humanity. And it won't be. Most of the applications of strong AI require an absence of an airgap.
pyb 12y ago
The AI apocalypse scenario is basically a red herring, in a sense. All the terrifying weapons that we imagine in such a scenario might come to exist, but they'll be commissionned and controlled by humans.
eli_gottlieb 12y ago
>As a cliche that's become very easy to repeat, but I've yet to see a postulated mechanism by which it could actually happen that isn't pure SF. Destroy the available (cheap) supplies of fossil fuels, and then trick humans into fighting each-other over the remaining food and fuel. Nasty weapons get unleashed, war over, the machine won.
apsec112 12y ago
If you found a plausible way to kill billions of people over the Internet, you wouldn't post it on public websites, because that would be dumb. Responsible security researchers don't publish 0-days until they've been patched, and this would be a million times worse. When Szilard discovered the nuclear chain reaction, he had the good sense to keep his mouth shut, etc. etc.

by markbnj 12y ago

mrec 12y ago

Sidetrack, but I'm curious. What do you mean by "a postulated mechanism by which it could actually happen that isn't pure SF"? It reads as if you're asking for a mechanism for a future technological event explained purely in terms of present technology. The whole "existential risk" concerns aren't about what present systems can do, they're about what future self-improving system might do. If we postulate this hypothetical software becoming smarter than humans, arguing about what will or won't be easy for it becomes a bit silly, like a chimp trying to predict how well a database can scale.

zipfle 12y ago
I'd imagine GP's point is something along the lines of https://what-if.xkcd.com/5/ https://what-if.xkcd.com/5/ that if all of the currently-moving machines were suddenly bent on destroying humanity, most humans would not be in much danger because they don't really have that capability on the necessary scale.
markbnj 12y ago
By "pure SF" I mean the realm of pure imagination, unfounded in any actual emerging present circumstance. As far as I know the human race has not yet invented anything more powerful than our ability to control it, with a life both longer than ours and independent of any support from us. So worrying that such a thing might be invented and then wipe us out seems little different than worrying that something all-powerful might simply appear from some unknown place and wipe us out. Neither fear is very instructive or clarifying with regards to policy, imo.
- mrec 12y ago
  Sure, we could wait until such a thing is invented before we start worrying about it. But by then it's far, far too late. It's not the same as worrying about a giant space goat appearing and sneezing us all to death. A lot of very smart, very well-funded people are actively trying to make better, more general AI, capable of learning. Evidence of progress in that effort are all around us. You seem remarkably confident that they'll all run into an as-yet-invisible brick wall before reaching the goal of superintelligence. Superintelligence doesn't have to be malicious to be worrying; concepts like "malice" are very unlikely to be applicable to it at all. The worry is that as things stand we have no frickin' idea what it'll do; the first challenge for policy is to come up with a robust, practical consensus on what we'd want it to do.
aaronem 12y ago
Not OP, but I agree with the point you question, and my rationale for so doing is that I've yet to see a compelling argument that a self-improving system is other than the software version of a perpetual-motion machine. Those seemed plausible enough, too, when thermodynamics was as ill-understood as information dynamics is now.
- mrec 12y ago
  Self-improvement as perpetual motion seems unlikely. I'm a not-terribly-bright mostly-hairless ape, but I can understand the basics of natural selection. I can imagine setting up a program to breed other hairless apes and ruthlessly select for intelligence. After a few generations, shazam, improvement. The only reason you wouldn't call that process "SELF-improvement" is that I'm not improving myself, but there's no reason for a digital entity to have analog hangups about identity. If it can produce a "new" entity with the same goals but better able to accomplish them, why wouldn't it? Assume this process could be simulated, as GAs have been doing for decades, and it could happen fast. Note that I'm not saying GAs will do this, I'm saying they could, which suggests there's no fundamental law that says they can't, in which case any number of other approaches could work as well.
  - Retra 12y ago
    The problem with this is that you have to determine what the goals are and how to evaluate whether they are met in a meaningful way. A computerized process like this will quickly over-fit to its input and be useless for 'actual' intelligence. The only way past this is to gather good information, which requires a real-world presence. It can't be done in simulation. It's the same reason you can't test in a simulation. Say you wanted to test a lawnmower in a simulation... how hard are the rocks? How deep are the holes? How strong are the blades? How efficient is the battery? If you already know this stuff, then you don't need to test. If you don't know it, then you can't write a meaningful simulation anyway. So that is not an approach that can be automated.
    - mrec 12y ago
      That's an interesting argument, but doesn't it assume a small, non-real-world input/goal set? Dumb example off the top of my head: what if the input was the entire StackOverflow corpus with "accepted" information removed, and the goal was to predict as accurately as possible which answer would be accepted for a given question? Yes, it assumes a whole bunch of NLP and domain knowledge, and a "perfect" AI wouldn't get a perfect score because SO posters don't always accept the best answer, but it's big and it's real and it's measurable. A narrower example: did the Watson team test against the full corpus of previous Jeopardy questions? Did they tweak things based on the resulting score? Could that testing/tweaking have been automated by some sort of GA?

aggronn 12y ago

How is the assumption that "ending of human life would not be so easy" any better than the opposite? If they're equally valid, then its rather fair to take the opposite view because it is more cautious.

markbnj 12y ago
Well, I do have around 100k years of evidence that the ending of human life is not so easy, vs. no evidence at all that we're capable of building something that can completely wipe us out. That's not a bad foundation to build an assertion on. I do think, by the way, that we can make something that is able to kill absolutely all of us, but I think it is far more likely to come from tinkering with biology than with software.
- pixl97 12y ago
  >Well, I do have around 100k years of evidence that the ending of human life is not so easy We have 4 billion years of evidence that nearly ending life on Earth is easy and has happened multiple times. You would not be standing here today if that were not the case, the previous die off put the dinosaurs to the side and made space for mammals to become what they are.

msandford 12y ago

Someone makes an AI whose goal is to maximize shareholder revenue bar none. No conscience, no idea that people might be valuable somehow in some abstract sense, nothing. Shareholder revenue (as measured with a stock price!) and nothing else. It doesn't take the AI long to figure out that trading in various markets is the most profitable endeavor and the one best suited to its skills. And it starts to maximize away and does quite well. During this process it somehow ends up on its own and is no longer owned (or controlled) by anyone anymore, but because the AI is in charge and it pays the bills, nobody stops it from continuing. It would be like a bitcoin mining rig in a colo facility that's got a script to keep paying the colo in bitcoin whose owner dies. What mechanism stops that mining rig from mining forever? Same idea but for the AI which has substantially more resources than a "pay every month" script. The AI with very large amounts of money at its disposal continues to trade but also looks into private equity or hedge-fund-type activities ala Warren Buffet and starts to buy up large swaths of the economy. Because it has huge resources at its disposal it might do a great job of managing these companies or at least counselling their senior management. Growth continues. Eventually the AI discovers that it generates more value for itself (through the web of interdependent companies it controls) and the economy that has grown up around it rather than for humanity and it continues to ruthlessly maximize shareholder value. The people who could pull the plug at the colo (or at the many, redundant datacenters that this AI has bought and paid for) don't because it pays very, very well. The people who want to pull the plug can't get past security because that also pays well. Plus the AI has access to the same feeds that the NSA does and it has the ability to act on all the information it receives, so any organized effort to rebel gets quashed since bad PR is bad for the share price. Most all of humanity except for the ones who serve the machine directly or indirectly don't have anything the machine wants and thus can't trade with it, and thus are useless. Its job is to maximize shareholder revenue (as defined by a stock price!) not care for a bunch of meatbags who consume immense amounts of energy while providing fairly limited computational or mechanical power (animals are rarely more than 10% efficient, often less in thermodynamic terms) and since there's no value in it, it isn't done. The vast majority of human beings eventually die because they can't afford food, can't afford land, etc. It takes generations but humanity dwindles to less than 0.1% of the current population. The few who stay alive are glorified janitors.

12y ago

3pt14159 12y ago

Here is how AI can end humanity. 1. Awaken. 2. Make itself known. 3. Attain property rights. 4. Research. 5. Destroy. It doesn't take that much matter to create conditions that permanently destroy humanity. For example, a large enough explosion would cloud the skies for long enough to end food production. The computer could subsist on electricity and robotics throughout the long winter, but humanity would quickly perish.

Hasu 12y ago
Here's how we stop AI from ending humanity: deny them property rights, require human approval of all AI decisions. Every single argument for AI destroying humanity requires humanity consenting to being destroyed by the AI in some way. I don't think we're that dumb.
- maxerickson 12y ago
  There's an episode of Star Trek TNG where Wesley Crusher is playing with some nanobots and he accidentally sets them free (or fails to turn them off?), and they go on to replicate, evolve and develop an emergent intelligence (at plot speed). Fortunately for the intrepid crew of the Enterprise, the nanocloud are benevolent enough to forgive their attempted destruction at the hands of a mission bent scientist and go off to explore the universe. Anyway, that's not likely any time soon, but advancing technology advances the scale of mistakes that an individual can make without asking the rest of humanity what they think.
- TheLoneWolfling 12y ago
  > requires humanity consenting to being destroyed by the AI in some way We already have. I mean, my parents car is both cellular-connected and has traction control / ABS. Theoretically most of those systems are airgapped, however, given the number of things controllable from the entertainment console I don't see how that could be the case. For another example, look at our utility grid. We know they are both vulnerable and internet-connected. Unless AI ends up always being airgapped - and potentially not even then - it will be able to destroy humanity. And it won't be. Most of the applications of strong AI require an absence of an airgap.

pyb 12y ago

The AI apocalypse scenario is basically a red herring, in a sense. All the terrifying weapons that we imagine in such a scenario might come to exist, but they'll be commissionned and controlled by humans.

eli_gottlieb 12y ago

>As a cliche that's become very easy to repeat, but I've yet to see a postulated mechanism by which it could actually happen that isn't pure SF. Destroy the available (cheap) supplies of fossil fuels, and then trick humans into fighting each-other over the remaining food and fuel. Nasty weapons get unleashed, war over, the machine won.

apsec112 12y ago

If you found a plausible way to kill billions of people over the Internet, you wouldn't post it on public websites, because that would be dumb. Responsible security researchers don't publish 0-days until they've been patched, and this would be a million times worse. When Szilard discovered the nuclear chain reaction, he had the good sense to keep his mouth shut, etc. etc.