The Yudkowskiites are all about coherent fantasies untested by reality, as are… really… a lot of philosophers throughout history. Maybe most. I fundamentally d

The Yudkowskiites are all about coherent fantasies untested by reality, as are… really… a lot of philosophers throughout history. Maybe most. I fundamentally do not believe in knowing without sensing or learning without experiencing. Of course it need not be direct experience. You can “download” information. But that information had to be gathered somehow at some point. There is only so much training data. As I said — I don’t dismiss the idea of AI challenging human intellect or replacing human jobs. An AI “merely” as smart as a human but tireless and faster could seem superhuman. I am just intensely skeptical of the idea of something learning to self improve and then magically taking off into some godlike superintelligence realm far beyond what is latent in or implied by its training data. That would be an informatic perpetual motion machine. It would, in fact, be magic, in the fantasy sense.