3 ms·

I'm always unimpressed by how certain people are that a hard takeoff is even possible. As near as we can tell, designing things is hard. (At least so long as th

I'm always unimpressed by how certain people are that a hard takeoff is even possible. As near as we can tell, designing things is hard. (At least so long as the P NP boundary exists.) I don't expect revisions of an AGI to have enough marginal intelligence over previous revisions to offset the increase in complexity that designing a further better one requires. It seems more likely to me that any given AGI will asymptote or grow logarithmically (rather than exponentially) simply because designing things seems to in general be exponentially hard relative to the number of states involved.

AngryParsley 16y ago
You don't have assign high probability to hard takeoff to support ethical/friendly AI research. The consequences of a hard takeoff are so huge (potentially destroying humanity) that even a low chance is worth making lower. Humans have been running on the same hardware for several thousand years. Just our improved knowledge and culture has been enough to keep us growing at an exponential rate. If our brains were end-user modifiable and had APIs, we'd be able to increase our growth rate even faster. An AI could do even better than self-modification. It could buy or hack enough computers to give itself thousands of times more processing power. I don't assign high probability to that sort of scenario, but it is high enough that it's worth ameliorating.
3pt14159 16y ago
Provided that the next increment is within grasp of the current increment for all increments, it will likely come very, very quickly. Remember, even logarithms hit infinity as x -> inf. I agree that the base intelligence of the AI would have to be at least within the realm of a very intelligent human to begin the cycle, but once begun, I'm unsure. As for asymptotes, if there are hard limits and structural changes that need to take place in order to break an IQ of, say, 220 then it stands to reason that we will never break it (or near-never, IQ as a system has no max or min, so if 10 trillion humans are eventually born then by definition we should have some way ahead of that measure). The reason we could never break that limit is that if we could break it the AI already would have, but since it is more intelligent than we are and can think orders of magnitudes faster than we can, it is unlikely that we would do so. Assuming that an AI has near instant thought processing time, it will either make the next level soon, or we never will.

AngryParsley 16y ago

You don't have assign high probability to hard takeoff to support ethical/friendly AI research. The consequences of a hard takeoff are so huge (potentially destroying humanity) that even a low chance is worth making lower. Humans have been running on the same hardware for several thousand years. Just our improved knowledge and culture has been enough to keep us growing at an exponential rate. If our brains were end-user modifiable and had APIs, we'd be able to increase our growth rate even faster. An AI could do even better than self-modification. It could buy or hack enough computers to give itself thousands of times more processing power. I don't assign high probability to that sort of scenario, but it is high enough that it's worth ameliorating.

3pt14159 16y ago

Provided that the next increment is within grasp of the current increment for all increments, it will likely come very, very quickly. Remember, even logarithms hit infinity as x -> inf. I agree that the base intelligence of the AI would have to be at least within the realm of a very intelligent human to begin the cycle, but once begun, I'm unsure. As for asymptotes, if there are hard limits and structural changes that need to take place in order to break an IQ of, say, 220 then it stands to reason that we will never break it (or near-never, IQ as a system has no max or min, so if 10 trillion humans are eventually born then by definition we should have some way ahead of that measure). The reason we could never break that limit is that if we could break it the AI already would have, but since it is more intelligent than we are and can think orders of magnitudes faster than we can, it is unlikely that we would do so. Assuming that an AI has near instant thought processing time, it will either make the next level soon, or we never will.