3 ms·

The singularity is not just about artificial general intelligence, but about AGI entering a positive feedback loop of intelligence amplification. I retain some

by jl6 5y ago

The singularity is not just about artificial general intelligence, but about AGI entering a positive feedback loop of intelligence amplification. I retain some skepticism that this is inevitable. The theory seems to rest on the idea that because AGI has a perfectly observable mechanical substrate, it will be possible to analyze and understand the fundamental causes of intelligence and thus improve upon it - a relatively simple matter of a software upgrade. This is unlike our own biological intelligence where we lack the ability to fully understand the state of the system - because of engineering limitations on observing the whole system at fine electron-microscope detail, physical limitations on observing possibly-quantum-scale behaviour, and also ethical limitations. But! The evidence we have from systems like GPT-3 is that even with perfect observability of the substrate, we find it extremely difficult to discern what exactly is causing the behaviour we see. While I don’t think super-AGI is impossible, I also think it’s still plausible that an entity’s intelligence emerges at a level of abstraction so high that it cannot be analyzed by that entity.

ephbit 5y ago
> The theory seems to rest on the idea that because AGI has a perfectly observable mechanical substrate, it will be possible to analyze and understand the fundamental causes of intelligence and thus improve upon it - a relatively simple matter of a software upgrade. I've never thought of it this way, that AGI will necessarily develop because it beyond some point picks up understanding about itself and then intentionally improves. I rather see it this way: the development of AGI is in the beginning rather unintentional. It will just be the result of a system dynamic that is more stable than other dynamics. Evolution isn't intentional either, is it? It's coincidence and randomness, trial and error, and dynamically more stable beings outlive other less stable beings. Stability appears to me to somewhat correlate with complexity. More complex beings having more ability to adapt and thus be able to survive in a greater set of conditions. As soon as the complexity of a beings information processing - which is mostly represented through number of brain cells and links (in biology) and number of artificial neurons (in AI) - goes into a dynamic of growth (like a positive feedback loop), I'd say it's pretty much on the pathway to AGI. Provided, the being isn't physically limited to a degree of complexity that's just too low.
- jl6 5y ago
  I’m not sure I properly understand what you are saying, but I think your idea is that 10x’ing the complexity of a system has the potential to support 10x the intelligence, and so the path to super-intelligence is not through deliberate intelligence-engineering but through scaling up the hardware and applying artificial or natural selection? Is that right?

by jl6 5y ago

ephbit 5y ago

> The theory seems to rest on the idea that because AGI has a perfectly observable mechanical substrate, it will be possible to analyze and understand the fundamental causes of intelligence and thus improve upon it - a relatively simple matter of a software upgrade. I've never thought of it this way, that AGI will necessarily develop because it beyond some point picks up understanding about itself and then intentionally improves. I rather see it this way: the development of AGI is in the beginning rather unintentional. It will just be the result of a system dynamic that is more stable than other dynamics. Evolution isn't intentional either, is it? It's coincidence and randomness, trial and error, and dynamically more stable beings outlive other less stable beings. Stability appears to me to somewhat correlate with complexity. More complex beings having more ability to adapt and thus be able to survive in a greater set of conditions. As soon as the complexity of a beings information processing - which is mostly represented through number of brain cells and links (in biology) and number of artificial neurons (in AI) - goes into a dynamic of growth (like a positive feedback loop), I'd say it's pretty much on the pathway to AGI. Provided, the being isn't physically limited to a degree of complexity that's just too low.

jl6 5y ago
I’m not sure I properly understand what you are saying, but I think your idea is that 10x’ing the complexity of a system has the potential to support 10x the intelligence, and so the path to super-intelligence is not through deliberate intelligence-engineering but through scaling up the hardware and applying artificial or natural selection? Is that right?