4 ms·

But what characterizes those other arrangements of matter? What is their energy consumption? What are their switching speeds? Can cognition grow unbounded, and

by atroyn 9y ago

But what characterizes those other arrangements of matter? What is their energy consumption? What are their switching speeds? Can cognition grow unbounded, and if so, why? Can any brain improve on its own process of cognition? If other arrangements of matter perform cognition equally well, why don't we see them in the independent evolution of life with wildly different structures for everything else? There is far too much woo and not enough rigorous thinking around the notion of AI superintelligence, singularity, or whatever other incantation people care to use.

sullyj3 9y ago
> why don't we see them in the independent evolution of life Because evolution is a very weak optimisation process. It has very limited materials to work with - everything has to be made of meat. The only changes it can make are incredibly indirect ones - mutations in DNA base pairs, leading to different proteins being manufactured or some such (I'm not a biologist, obviously) leading to subtle changes in the makeup of the resulting organism. Can you imagine how difficult engineering would be through a medium like that? It'd be like trying to perform brain surgery with a sledgehammer. And it's not even purposefully optimising for anything, let alone intelligence. The changes all happen completely at random, and the ones that manage to reproduce stick around. Moreover, the fact that such a weak process as natural selection can produce something as smart as us, should really make you consider that humans, with optimisation power far superior to natural selection, can produce something far smarter.
sullyj3 9y ago
In regards to self improvement, my argument wasn't really about that. I was trying to demonstrate the viability of at least human level intelligence. I don't know how likely intelligence explosion scenarios are. The possibility of an AI of human level intelligence being able to improve itself seems obviously plausible - if humans can engineer intelligence, then human level AI can. Whether it can do that rapidly is far more questionable. However, even if you think the likelihood of fast improvement cycles is quite low, it still warrants some serious thought, given the size of the potential payoffs and downsides.

by atroyn 9y ago

sullyj3 9y ago

> why don't we see them in the independent evolution of life Because evolution is a very weak optimisation process. It has very limited materials to work with - everything has to be made of meat. The only changes it can make are incredibly indirect ones - mutations in DNA base pairs, leading to different proteins being manufactured or some such (I'm not a biologist, obviously) leading to subtle changes in the makeup of the resulting organism. Can you imagine how difficult engineering would be through a medium like that? It'd be like trying to perform brain surgery with a sledgehammer. And it's not even purposefully optimising for anything, let alone intelligence. The changes all happen completely at random, and the ones that manage to reproduce stick around. Moreover, the fact that such a weak process as natural selection can produce something as smart as us, should really make you consider that humans, with optimisation power far superior to natural selection, can produce something far smarter.

sullyj3 9y ago

In regards to self improvement, my argument wasn't really about that. I was trying to demonstrate the viability of at least human level intelligence. I don't know how likely intelligence explosion scenarios are. The possibility of an AI of human level intelligence being able to improve itself seems obviously plausible - if humans can engineer intelligence, then human level AI can. Whether it can do that rapidly is far more questionable. However, even if you think the likelihood of fast improvement cycles is quite low, it still warrants some serious thought, given the size of the potential payoffs and downsides.