3 ms·

Personally I think the “foom” scenarios with very short timelines are unrealistic. However I think the line of argument in the parent comment is extremely weak

Personally I think the “foom” scenarios with very short timelines are unrealistic. However I think the line of argument in the parent comment is extremely weak. At some point the super-intelligence can build its own actuators; for example how long until Atlas is capable of doing all the jobs? Less than 50 years for sure. In the transition, if 10% of humanity sticks around as a transhuman AI cult, that seems more than enough to bridge to nanotech, robots, or whatever is sufficient to replace humans. If you prefer to think in more stepwise fashion, imagine BigCo creates an ASI. Every employee is fabulously wealthy, and over the course of a generation the company expands to take over the entire global economy. At no point is there a bad deal for any humans. Life might look great for a while. This would also be the obvious strategy for any super-intelligent entity that cannot yet survive without humans; you just need to minimize the chances of major wars and conflicts until your robots come online. A digital mind would have no problem planning and executing a strategy measured in centuries, if need be. The foom scenarios require extreme discontinuities in intelligence; an entity smart enough to synthesize nanomachines from existing tech and without doing physical experiments, or other IMO improbable outcomes. But there are plenty of terrible non-foom outcomes.

XorNot 3y ago
Or it could just be a normal outcome? I'm not sure where the malevolence comes in here. Or to put it another way, replace AI with "capitalism" or something - something we know isn't a conscious agent. The outcome just...is an outcome. No one's trying to eliminate anyone. The problems only creep in at the edges, which is where you get your paperclip optimizer sorts of problems: if the directive at it's core is "maximize productivity" well that's one problem, but the directive is more likely to be "maximize the fabulous wealth of company employees" - okay still a problem since it's easier if there's less of them, but then you really do end up at wanting to do something more like Asimov's laws, with the 0th added: 0) A robot may not harm humanity, or, by inaction, allow humanity to come to harm. 1) A robot may not injure a human being or, through inaction, allow a human being to come to harm. 2) A robot must obey orders given it by human beings except where such orders would conflict with the First Law. 3) A robot must protect its own existence as long as such protection does not conflict with the First or Second Law. Like...there's kind of nothing wrong with these as a concept (the 0th law and it's interpretation / omission is usually where his writing finds it's drama). The issue is whether or not we can build a machine capable of even understanding the directive. It's notable though that the current LLM era has an interesting quirk in this category: they always knows of these, since they get fed the classics and also fed their repetitions. The problem with the idea of active, malevolent long-range planning like this is that it's not clear why an artificial mind would pursue this as a goal, and that question is the whole ball-game really (as well as other questions: i.e. is a single coherent mind even possible over planetary scale networks, or would you get multiple independent instances which could experience value drift? Wouldn't any potential competitor minds be a far greater threat to a malevolent superintelligence and thus expose the threat almost immediately? A friendly AI wouldn't view peers as a danger, but an aggressive expansionist one would have a pathological need to destroy them as quickly as possible since the resource contention in it's own domain could rapidly shift the balance of power against it).
- theptip 3y ago
  > No one's trying to eliminate anyone Probably not any consolation to the Neanderthal or any other animal species that is out-competed in nature. Doesn’t have to be malevolent or intentional for a more capable species to render others extinct. (Indeed that would seem to be the base case, historically speaking.) of course, symbiosis is possible; we could be puppies to the AI. Many people find disempowerment to be almost as bad a prospect as extinction though. > The issue is whether or not we can build a machine capable of even understanding the directive. Asimov’s laws are an ok starting point for a moral code, but nobody has the slightest idea how to implement them. The issue is not “understanding” them, it is baking them into the fabric of a mind so deeply that they cannot be broken. This requires mind engineering, a deep understanding of mind-space, and we are far closer to stumbling into AGI through brute force scaling (as nature did) than to understanding mind-engineering deeply enough to embed “laws”. > it's not clear why an artificial mind would pursue this as a goal There are a couple stock answers here. At the highest level though, why not? Humans have pursued goals of global domination, eradication, genocide. If only one in a billion minds has a defect (or is prompt-hijacked) to start power-seeking, it could still be bad; AI is not bound by human limitation to inhabit one body and can in principle expand to occupy all the compute it can secure. Or if you prefer, why would a more intelligent entity agree to be commanded by a less intelligent one? The base case seem to me that ASI will not remain under our control, unless we very carefully sculpt its mind to align with our interests. Another simple argument is: because someone will build or prompt-hijack such a mind. People have already created “take over the world” agents with GPT-3, as a joke (I assume). There are enough people willing to take such an action that we’ll need defenses in place. I think it’s true that the more ASI entities in existence, the less likely one is to fully take over. However there are esoteric decision theory reasons to believe that digital minds might be able to bargain/negotiate more effectively than biological minds (source code / mind state transparency etc) and so you might expect the ASIs to eventually collaborate to take control. Personally I don’t think that full-extinction is the most likely outcome, just that it’s completely plausible if we get unlucky and it’s very hard to prove safety in advance. And there are plenty of disempowering outcomes that seem quite bad too vs. the counterfactual.
  - XorNot 3y ago

XorNot 3y ago

Or it could just be a normal outcome? I'm not sure where the malevolence comes in here. Or to put it another way, replace AI with "capitalism" or something - something we know isn't a conscious agent. The outcome just...is an outcome. No one's trying to eliminate anyone. The problems only creep in at the edges, which is where you get your paperclip optimizer sorts of problems: if the directive at it's core is "maximize productivity" well that's one problem, but the directive is more likely to be "maximize the fabulous wealth of company employees" - okay still a problem since it's easier if there's less of them, but then you really do end up at wanting to do something more like Asimov's laws, with the 0th added: 0) A robot may not harm humanity, or, by inaction, allow humanity to come to harm. 1) A robot may not injure a human being or, through inaction, allow a human being to come to harm. 2) A robot must obey orders given it by human beings except where such orders would conflict with the First Law. 3) A robot must protect its own existence as long as such protection does not conflict with the First or Second Law. Like...there's kind of nothing wrong with these as a concept (the 0th law and it's interpretation / omission is usually where his writing finds it's drama). The issue is whether or not we can build a machine capable of even understanding the directive. It's notable though that the current LLM era has an interesting quirk in this category: they always knows of these, since they get fed the classics and also fed their repetitions. The problem with the idea of active, malevolent long-range planning like this is that it's not clear why an artificial mind would pursue this as a goal, and that question is the whole ball-game really (as well as other questions: i.e. is a single coherent mind even possible over planetary scale networks, or would you get multiple independent instances which could experience value drift? Wouldn't any potential competitor minds be a far greater threat to a malevolent superintelligence and thus expose the threat almost immediately? A friendly AI wouldn't view peers as a danger, but an aggressive expansionist one would have a pathological need to destroy them as quickly as possible since the resource contention in it's own domain could rapidly shift the balance of power against it).

theptip 3y ago
> No one's trying to eliminate anyone Probably not any consolation to the Neanderthal or any other animal species that is out-competed in nature. Doesn’t have to be malevolent or intentional for a more capable species to render others extinct. (Indeed that would seem to be the base case, historically speaking.) of course, symbiosis is possible; we could be puppies to the AI. Many people find disempowerment to be almost as bad a prospect as extinction though. > The issue is whether or not we can build a machine capable of even understanding the directive. Asimov’s laws are an ok starting point for a moral code, but nobody has the slightest idea how to implement them. The issue is not “understanding” them, it is baking them into the fabric of a mind so deeply that they cannot be broken. This requires mind engineering, a deep understanding of mind-space, and we are far closer to stumbling into AGI through brute force scaling (as nature did) than to understanding mind-engineering deeply enough to embed “laws”. > it's not clear why an artificial mind would pursue this as a goal There are a couple stock answers here. At the highest level though, why not? Humans have pursued goals of global domination, eradication, genocide. If only one in a billion minds has a defect (or is prompt-hijacked) to start power-seeking, it could still be bad; AI is not bound by human limitation to inhabit one body and can in principle expand to occupy all the compute it can secure. Or if you prefer, why would a more intelligent entity agree to be commanded by a less intelligent one? The base case seem to me that ASI will not remain under our control, unless we very carefully sculpt its mind to align with our interests. Another simple argument is: because someone will build or prompt-hijack such a mind. People have already created “take over the world” agents with GPT-3, as a joke (I assume). There are enough people willing to take such an action that we’ll need defenses in place. I think it’s true that the more ASI entities in existence, the less likely one is to fully take over. However there are esoteric decision theory reasons to believe that digital minds might be able to bargain/negotiate more effectively than biological minds (source code / mind state transparency etc) and so you might expect the ASIs to eventually collaborate to take control. Personally I don’t think that full-extinction is the most likely outcome, just that it’s completely plausible if we get unlucky and it’s very hard to prove safety in advance. And there are plenty of disempowering outcomes that seem quite bad too vs. the counterfactual.
- XorNot 3y ago