5 ms·

Sidetrack, but I'm curious. What do you mean by "a postulated mechanism by which it could actually happen that isn't pure SF"? It reads as if you're asking for

by mrec 12y ago

Sidetrack, but I'm curious. What do you mean by "a postulated mechanism by which it could actually happen that isn't pure SF"? It reads as if you're asking for a mechanism for a future technological event explained purely in terms of present technology. The whole "existential risk" concerns aren't about what present systems can do, they're about what future self-improving system might do. If we postulate this hypothetical software becoming smarter than humans, arguing about what will or won't be easy for it becomes a bit silly, like a chimp trying to predict how well a database can scale.

zipfle 12y ago
I'd imagine GP's point is something along the lines of https://what-if.xkcd.com/5/ https://what-if.xkcd.com/5/ that if all of the currently-moving machines were suddenly bent on destroying humanity, most humans would not be in much danger because they don't really have that capability on the necessary scale.
markbnj 12y ago
By "pure SF" I mean the realm of pure imagination, unfounded in any actual emerging present circumstance. As far as I know the human race has not yet invented anything more powerful than our ability to control it, with a life both longer than ours and independent of any support from us. So worrying that such a thing might be invented and then wipe us out seems little different than worrying that something all-powerful might simply appear from some unknown place and wipe us out. Neither fear is very instructive or clarifying with regards to policy, imo.
- mrec 12y ago
  Sure, we could wait until such a thing is invented before we start worrying about it. But by then it's far, far too late. It's not the same as worrying about a giant space goat appearing and sneezing us all to death. A lot of very smart, very well-funded people are actively trying to make better, more general AI, capable of learning. Evidence of progress in that effort are all around us. You seem remarkably confident that they'll all run into an as-yet-invisible brick wall before reaching the goal of superintelligence. Superintelligence doesn't have to be malicious to be worrying; concepts like "malice" are very unlikely to be applicable to it at all. The worry is that as things stand we have no frickin' idea what it'll do; the first challenge for policy is to come up with a robust, practical consensus on what we'd want it to do.
aaronem 12y ago
Not OP, but I agree with the point you question, and my rationale for so doing is that I've yet to see a compelling argument that a self-improving system is other than the software version of a perpetual-motion machine. Those seemed plausible enough, too, when thermodynamics was as ill-understood as information dynamics is now.
- mrec 12y ago
  Self-improvement as perpetual motion seems unlikely. I'm a not-terribly-bright mostly-hairless ape, but I can understand the basics of natural selection. I can imagine setting up a program to breed other hairless apes and ruthlessly select for intelligence. After a few generations, shazam, improvement. The only reason you wouldn't call that process "SELF-improvement" is that I'm not improving myself, but there's no reason for a digital entity to have analog hangups about identity. If it can produce a "new" entity with the same goals but better able to accomplish them, why wouldn't it? Assume this process could be simulated, as GAs have been doing for decades, and it could happen fast. Note that I'm not saying GAs will do this, I'm saying they could, which suggests there's no fundamental law that says they can't, in which case any number of other approaches could work as well.
  - Retra 12y ago
    The problem with this is that you have to determine what the goals are and how to evaluate whether they are met in a meaningful way. A computerized process like this will quickly over-fit to its input and be useless for 'actual' intelligence. The only way past this is to gather good information, which requires a real-world presence. It can't be done in simulation. It's the same reason you can't test in a simulation. Say you wanted to test a lawnmower in a simulation... how hard are the rocks? How deep are the holes? How strong are the blades? How efficient is the battery? If you already know this stuff, then you don't need to test. If you don't know it, then you can't write a meaningful simulation anyway. So that is not an approach that can be automated.
    - mrec 12y ago
      That's an interesting argument, but doesn't it assume a small, non-real-world input/goal set? Dumb example off the top of my head: what if the input was the entire StackOverflow corpus with "accepted" information removed, and the goal was to predict as accurately as possible which answer would be accepted for a given question? Yes, it assumes a whole bunch of NLP and domain knowledge, and a "perfect" AI wouldn't get a perfect score because SO posters don't always accept the best answer, but it's big and it's real and it's measurable. A narrower example: did the Watson team test against the full corpus of previous Jeopardy questions? Did they tweak things based on the resulting score? Could that testing/tweaking have been automated by some sort of GA?

by mrec 12y ago

zipfle 12y ago

I'd imagine GP's point is something along the lines of https://what-if.xkcd.com/5/ https://what-if.xkcd.com/5/ that if all of the currently-moving machines were suddenly bent on destroying humanity, most humans would not be in much danger because they don't really have that capability on the necessary scale.

markbnj 12y ago

By "pure SF" I mean the realm of pure imagination, unfounded in any actual emerging present circumstance. As far as I know the human race has not yet invented anything more powerful than our ability to control it, with a life both longer than ours and independent of any support from us. So worrying that such a thing might be invented and then wipe us out seems little different than worrying that something all-powerful might simply appear from some unknown place and wipe us out. Neither fear is very instructive or clarifying with regards to policy, imo.

mrec 12y ago
Sure, we could wait until such a thing is invented before we start worrying about it. But by then it's far, far too late. It's not the same as worrying about a giant space goat appearing and sneezing us all to death. A lot of very smart, very well-funded people are actively trying to make better, more general AI, capable of learning. Evidence of progress in that effort are all around us. You seem remarkably confident that they'll all run into an as-yet-invisible brick wall before reaching the goal of superintelligence. Superintelligence doesn't have to be malicious to be worrying; concepts like "malice" are very unlikely to be applicable to it at all. The worry is that as things stand we have no frickin' idea what it'll do; the first challenge for policy is to come up with a robust, practical consensus on what we'd want it to do.

aaronem 12y ago

Not OP, but I agree with the point you question, and my rationale for so doing is that I've yet to see a compelling argument that a self-improving system is other than the software version of a perpetual-motion machine. Those seemed plausible enough, too, when thermodynamics was as ill-understood as information dynamics is now.

mrec 12y ago
Self-improvement as perpetual motion seems unlikely. I'm a not-terribly-bright mostly-hairless ape, but I can understand the basics of natural selection. I can imagine setting up a program to breed other hairless apes and ruthlessly select for intelligence. After a few generations, shazam, improvement. The only reason you wouldn't call that process "SELF-improvement" is that I'm not improving myself, but there's no reason for a digital entity to have analog hangups about identity. If it can produce a "new" entity with the same goals but better able to accomplish them, why wouldn't it? Assume this process could be simulated, as GAs have been doing for decades, and it could happen fast. Note that I'm not saying GAs will do this, I'm saying they could, which suggests there's no fundamental law that says they can't, in which case any number of other approaches could work as well.
- Retra 12y ago
  The problem with this is that you have to determine what the goals are and how to evaluate whether they are met in a meaningful way. A computerized process like this will quickly over-fit to its input and be useless for 'actual' intelligence. The only way past this is to gather good information, which requires a real-world presence. It can't be done in simulation. It's the same reason you can't test in a simulation. Say you wanted to test a lawnmower in a simulation... how hard are the rocks? How deep are the holes? How strong are the blades? How efficient is the battery? If you already know this stuff, then you don't need to test. If you don't know it, then you can't write a meaningful simulation anyway. So that is not an approach that can be automated.
  - mrec 12y ago
    That's an interesting argument, but doesn't it assume a small, non-real-world input/goal set? Dumb example off the top of my head: what if the input was the entire StackOverflow corpus with "accepted" information removed, and the goal was to predict as accurately as possible which answer would be accepted for a given question? Yes, it assumes a whole bunch of NLP and domain knowledge, and a "perfect" AI wouldn't get a perfect score because SO posters don't always accept the best answer, but it's big and it's real and it's measurable. A narrower example: did the Watson team test against the full corpus of previous Jeopardy questions? Did they tweak things based on the resulting score? Could that testing/tweaking have been automated by some sort of GA?