4 ms·

I don't claim they are impossible (although I personally believe 5 is extremely unrealistic and 4 is difficult), I just have not seen any mathematical or experi

by golol 3y ago

I don't claim they are impossible (although I personally believe 5 is extremely unrealistic and 4 is difficult), I just have not seen any mathematical or experimental evidence. Experimental evidence would be an example of n accelerated self-imporvement process in a simpler model. Of course this is unlikely to happen, as one expect this kind of self-improvement to happen once a critical amount of intelligence is reached, and such examples should be impossible nowadays. Nevertheless, no experimental evidence. Mathematical evidence would be some kind of mathematical argument that suggests that a rapid growth of intelligence is plausible. Again, that is extremely high of me to expect as we don't really have any idea how to define these things mathematically. So I am setting the bar very high again. Okay, so while there is no experimental or mathematical evidence for rapid self-improvement, one may say that this is unrealistic to expect. But given the absence of positive evidence, let's look at negative evidence. Every growth pattern we see in machine learning is either some form of decaying growth, logarithmic or sigmoid shaped. The rapid self-improvement of superintelligence would be the first accelerating performance curve we see... >recursive self improvement is just a tighter loop than natural selection, biological reproduction and cellular growth. Just because some process is self-improving, doesn't mean it will grow at increasing speed. It may just converge to an asymptote. Or grow logarithmically. Also, the claimed self-improvement is so different from evolution that it's not great to think of it as "evolution but faster" imo.

nh23423fefe 3y ago
I can't tell if this is just a matter of different assumptions. I think superintelligence is obvious even in principle. I would analogize with some technology like nano-tech. The tech-tree starts with fire, metallurgy, chemistry, atomic physics, solid-state electronics and will eventually get to https://en.wikipedia.org/wiki/Atomically_precise_manufacturing https://en.wikipedia.org/wiki/Atomically_precise_manufacturi... There is nothing "in principle" to stop this trend, and even more than that, there is no limitation when constructing arrangements of matter outside physics and knowledge. You could accuse me of technological determinism if you wanted. superintelligence isnt something different than intelligence to me. Its just a transition in the substrate. Intelligence is the emergent capacity of a system to alter its environment to achieve goals. superintelligent systems are those with more capability. the main difference is the magnitude of the ability to correlate the system's internal model with external reality and then act on that model to produce low error predictions of the world state. the act-perceive-optimize loop of synthetic minds is unbounded in ways that biological minds are. artificial minds will build larger models of the world that correlate better with reality and they will search-think-tinker with their internal models to better achieve their goals. What could stop this? Sure you can posit an asymphtote. But what if i said the singularity creates minds that are 300 IQ? Smarter than any person would ever be even if trillions of humans were born. What if the number is 3000 IQ? I do think of it as evolution but faster because evolution isn't directed, its just global search optimization. AI will be driven toward capability, not optimal fitness.
- golol 3y ago
  You know I agree that superintelligence is totally possible, but what I disagree with is that it follows immediately after AGI. Suppose you reach the level of AGI. Then your AGI can not just write down the weights for a smarter AI. It has roughly the intelligence of humans, so to build a better AI it would have to do something similar to what humans do: Large scale training on large computers . This takes time and is easily supervisable. I don't see a risk of rapid singularity like intelligence blow up. There is a risk of the AGI escaping but since it needs aignificant amounts of compute, and does mot have the intelligence to just rewrite itself into a decentralized abstract form, I again don't see a big danger. When we are at this stage we can start worrying about superintelligence, as well as AI rights and digital citzenship obviously.

by golol 3y ago

nh23423fefe 3y ago

I can't tell if this is just a matter of different assumptions. I think superintelligence is obvious even in principle. I would analogize with some technology like nano-tech. The tech-tree starts with fire, metallurgy, chemistry, atomic physics, solid-state electronics and will eventually get to https://en.wikipedia.org/wiki/Atomically_precise_manufacturing https://en.wikipedia.org/wiki/Atomically_precise_manufacturi... There is nothing "in principle" to stop this trend, and even more than that, there is no limitation when constructing arrangements of matter outside physics and knowledge. You could accuse me of technological determinism if you wanted. superintelligence isnt something different than intelligence to me. Its just a transition in the substrate. Intelligence is the emergent capacity of a system to alter its environment to achieve goals. superintelligent systems are those with more capability. the main difference is the magnitude of the ability to correlate the system's internal model with external reality and then act on that model to produce low error predictions of the world state. the act-perceive-optimize loop of synthetic minds is unbounded in ways that biological minds are. artificial minds will build larger models of the world that correlate better with reality and they will search-think-tinker with their internal models to better achieve their goals. What could stop this? Sure you can posit an asymphtote. But what if i said the singularity creates minds that are 300 IQ? Smarter than any person would ever be even if trillions of humans were born. What if the number is 3000 IQ? I do think of it as evolution but faster because evolution isn't directed, its just global search optimization. AI will be driven toward capability, not optimal fitness.

golol 3y ago
You know I agree that superintelligence is totally possible, but what I disagree with is that it follows immediately after AGI. Suppose you reach the level of AGI. Then your AGI can not just write down the weights for a smarter AI. It has roughly the intelligence of humans, so to build a better AI it would have to do something similar to what humans do: Large scale training on large computers . This takes time and is easily supervisable. I don't see a risk of rapid singularity like intelligence blow up. There is a risk of the AGI escaping but since it needs aignificant amounts of compute, and does mot have the intelligence to just rewrite itself into a decentralized abstract form, I again don't see a big danger. When we are at this stage we can start worrying about superintelligence, as well as AI rights and digital citzenship obviously.