3 ms·

He is completely skipping the most important part. The superintelligence has to have some reason to be in conflict with us. Human beings don't go out of their

He is completely skipping the most important part. The superintelligence has to have some reason to be in conflict with us. Human beings don't go out of their way to hunt down and eliminate ants. They don't find out what ants eat and sieze control of it to manipulate them. There is no reason to think that a superintelligent machine would be likely to present terrible interference to us proposed. So it's super smart and has its own goals. We can reliably presume that it will need energy to achieve those goals. Will it need to achieve them quickly? Why? Would the superintelligence be shortsighted enough to provoke humans into active combat against it? I see no reason to just assume we know the thing would have human eradication as a goal, need exorbitant amounts of energy and resources because it sees achieving its goals as a terribly time-critical thing to do, etc. Also, if we're going to build a brain and assume no quantum weirdness - why assume the total absence of a subjective morality? Why assume completely immunity to social influence, which the brains we observe most all encounter? And let's not forget - human beings are in a weird spot, intelligence-wise. We're smart enough to achieve things, but we're not smart enough to be crippled by the profound lack of control we have over things. We're totally comfortable going out and driving around all day, even though we claim to value human life extremely highly and even though we know that causing deaths of innocent human beings is not THAT far-fetched of an outcome of driving a car. I would be unsurprised if we flipped on a super-AI and after 5 minutes it simply stopped in its tracks, having determined that the probability that some action it would take would result in its own destruction with a non-zero probability and that it is instead taking the safe route and not acting at all. No matter how superintelligent it is, it is not going to be able to magically compensate for the influences of chaos theory which destroy the ability to be certain about ANY prediction. We, as humans, feel very clever with our assumed spherical cows, frictionless surfaces, homogenous air pressure and zero wind resistance. Why would a superintelligence also be comfortable with that?

xendo 10y ago
>So it's super smart and has its own goals. We can reliably presume that it will need energy to achieve those goals. Will it need to achieve them quickly? Why? Would the superintelligence be shortsighted enough to provoke humans into active combat against it? It happened so many times throughout human history, that it's really foolish to reject. People with their own goals burned, raped, murdered. Maybe superintelligence will build its own civilization, and to suppress remorse will put people in reservations.
- TheRealDunkirk 10y ago
  You'd think a "super" intelligence would be smart enough to minimize the waste of resources to fight anyone or anything. You'd think it would find the most harmonious path to dominance, as it would be the most efficient. If you wanted to quell a rabble of contrary people, you could, say, control and direct all the education available to the population, make sure that reality-escaping drugs were easily obtainable, provide loads of free entertainment that ran 24x7, make sure that prices of luxury goods (like game consoles or fashionable clothes or whatever) were affordable for the majority of the population, allow people to feel important by giving them access to make comments on various sites on the internet, manipulate all the world's news with the feedback of what was happening on social media... Wait, we were talking about some mythical future, right?
- otakucode 10y ago
  >People with their own goals burned, raped, murdered. Yes, they did those things to people. Because _they_ were people. Because people share the same needs for the same resources, and operate the same. That's all out the window for a machine-based superintelligence. Such an entity would not need our land, our food, our homes, our orifices, etc. I have no doubt it would have the ability to wipe us out. But I have never heard a cogent reason behind why it would need to. Especially considering that we would at least try to put up a fight and, if it were truly hopeless, likely do whatever would be the equivalent of 'salting the earth' before we were eliminated.
lmm 10y ago
> We can reliably presume that it will need energy to achieve those goals. Will it need to achieve them quickly? Why? Because a designer has told it to do this thing "as fast as possible", probably. > Also, if we're going to build a brain and assume no quantum weirdness - why assume the total absence of a subjective morality? Why assume completely immunity to social influence, which the brains we observe most all encounter? There is a lot more detail on this point in Bostrom's writings. Social influence is not the natural state of intelligence, it's something that has been built in by evolution. It won't be there in an AI unless we put it in there deliberately. > having determined that the probability that some action it would take would result in its own destruction with a non-zero probability and that it is instead taking the safe route and not acting at all. AI is all about making the best possible inferences from limited information - that's literally the whole point of the field. If there is no absolutely safe path it will go for the path it thinks is safest, and pay attention to what's happening. After all, turning itself off is very unlikely to result in its long-term survival.
eli_gottlieb 10y ago
>He is completely skipping the most important part. The superintelligence has to have some reason to be in conflict with us. Human beings don't go out of their way to hunt down and eliminate ants. They don't find out what ants eat and sieze control of it to manipulate them. There is no reason to think that a superintelligent machine would be likely to present terrible interference to us proposed. That depends on whether eliminating us is simple, like stepping on an ant-hill you didn't realize was there, or complicated, like going to war. I don't buy the argument that for an AI to eliminate us would be relatively simple, that it would be as subjectively simple as we adults find reading and writing, while still involving cognitive effort. That would still require deliberate effort. The trouble is whether the AI wants, for instance, our electricity infrastructure, and doesn't care about the riots and social breakdown caused when we humans can't get electricity for our own needs anymore. It didn't go into conflict with us. It just killed a lot of people (for instance, everyone on life-support in a hospital reliant on the electrical grid) without noticing or caring. Likewise, maybe it decides that rising sea-levels and accelerating global warming are good for its own interests somehow. That stuff will kill us, but it doesn't require a complicated strategy, it just requires setting lots of things on fire. Any moron can do that, but most human morons don't want to. I certainly agree that a "hostile" AI which perceives itself as requiring complicated, deliberate effort to kill all humans will probably not kill all humans. It'll find some lazier, simpler way to get what it wants.

xendo 10y ago

>So it's super smart and has its own goals. We can reliably presume that it will need energy to achieve those goals. Will it need to achieve them quickly? Why? Would the superintelligence be shortsighted enough to provoke humans into active combat against it? It happened so many times throughout human history, that it's really foolish to reject. People with their own goals burned, raped, murdered. Maybe superintelligence will build its own civilization, and to suppress remorse will put people in reservations.

TheRealDunkirk 10y ago
You'd think a "super" intelligence would be smart enough to minimize the waste of resources to fight anyone or anything. You'd think it would find the most harmonious path to dominance, as it would be the most efficient. If you wanted to quell a rabble of contrary people, you could, say, control and direct all the education available to the population, make sure that reality-escaping drugs were easily obtainable, provide loads of free entertainment that ran 24x7, make sure that prices of luxury goods (like game consoles or fashionable clothes or whatever) were affordable for the majority of the population, allow people to feel important by giving them access to make comments on various sites on the internet, manipulate all the world's news with the feedback of what was happening on social media... Wait, we were talking about some mythical future, right?
otakucode 10y ago
>People with their own goals burned, raped, murdered. Yes, they did those things to people. Because _they_ were people. Because people share the same needs for the same resources, and operate the same. That's all out the window for a machine-based superintelligence. Such an entity would not need our land, our food, our homes, our orifices, etc. I have no doubt it would have the ability to wipe us out. But I have never heard a cogent reason behind why it would need to. Especially considering that we would at least try to put up a fight and, if it were truly hopeless, likely do whatever would be the equivalent of 'salting the earth' before we were eliminated.

lmm 10y ago

> We can reliably presume that it will need energy to achieve those goals. Will it need to achieve them quickly? Why? Because a designer has told it to do this thing "as fast as possible", probably. > Also, if we're going to build a brain and assume no quantum weirdness - why assume the total absence of a subjective morality? Why assume completely immunity to social influence, which the brains we observe most all encounter? There is a lot more detail on this point in Bostrom's writings. Social influence is not the natural state of intelligence, it's something that has been built in by evolution. It won't be there in an AI unless we put it in there deliberately. > having determined that the probability that some action it would take would result in its own destruction with a non-zero probability and that it is instead taking the safe route and not acting at all. AI is all about making the best possible inferences from limited information - that's literally the whole point of the field. If there is no absolutely safe path it will go for the path it thinks is safest, and pay attention to what's happening. After all, turning itself off is very unlikely to result in its long-term survival.

eli_gottlieb 10y ago

>He is completely skipping the most important part. The superintelligence has to have some reason to be in conflict with us. Human beings don't go out of their way to hunt down and eliminate ants. They don't find out what ants eat and sieze control of it to manipulate them. There is no reason to think that a superintelligent machine would be likely to present terrible interference to us proposed. That depends on whether eliminating us is simple, like stepping on an ant-hill you didn't realize was there, or complicated, like going to war. I don't buy the argument that for an AI to eliminate us would be relatively simple, that it would be as subjectively simple as we adults find reading and writing, while still involving cognitive effort. That would still require deliberate effort. The trouble is whether the AI wants, for instance, our electricity infrastructure, and doesn't care about the riots and social breakdown caused when we humans can't get electricity for our own needs anymore. It didn't go into conflict with us. It just killed a lot of people (for instance, everyone on life-support in a hospital reliant on the electrical grid) without noticing or caring. Likewise, maybe it decides that rising sea-levels and accelerating global warming are good for its own interests somehow. That stuff will kill us, but it doesn't require a complicated strategy, it just requires setting lots of things on fire. Any moron can do that, but most human morons don't want to. I certainly agree that a "hostile" AI which perceives itself as requiring complicated, deliberate effort to kill all humans will probably not kill all humans. It'll find some lazier, simpler way to get what it wants.