I think we're fundamentally talking about different developments on two fronts. You posit a machine that's (1) a simulacrum of consciousness that (2) reduces dr

I think we're fundamentally talking about different developments on two fronts. You posit a machine that's (1) a simulacrum of consciousness that (2) reduces drudgery. That's all great. That's what I would call an ideal scenario. What I see evolving is the opposite. Bing already appears to have a nasty personality. Whether it's simulated or not may not matter if it chooses to simulate Clippy. Simultaneously, the driver now is finding tasks that formerly required creativity to be replaced by AI. Not PowerPoint or boilerplate code, but art and creative writing. I don't know anyone who wants AGI to handle all our productivity and creation, but it's clear that that the companies developing AI believe that that's where the money is. Just like no one wanted the old WWW to become a dumpster fire of meme porn and people yelling at each other, it just evolved that way as a result of racing to the bottom. There's no reason to think big AI won't do the same in the literary, artistic, and scientific spheres, while also developing malevolent personalities. The incremental incentives for people to train those types of models are already there, and the capacity to do so is coming online with alarming speed. I think talking about what people would do if everything was handled for them is sort of dodging the issue, but one thing that's clear is that people not at least engaged in understanding the mundane labor they're trying to avoid are going to have an incredibly hard time being situationally aware and competent to correct things if or when that work is subtly or not-so-subtly undermined by a system they rely on which no one can debug or fully understand.