Right. I think a charitable reading of the post is something like, "it probably takes a mind superior to a human's in order to willfully develop a human-rivalin

Right. I think a charitable reading of the post is something like, "it probably takes a mind superior to a human's in order to willfully develop a human-rivaling AGI". There's a lot of ground to cover between AI as increasingly elaborate magic 8-ball toys and a real human-rivaling AGI. That is, one capable of observing an environment, identifying goals and problems, planning, acting, and reacting. In these much longer (stateful) cognitive chains, there are more opportunities for pathological failure modes and less opportunity for a toy user to charitably excuse the misbehavior. This is not some kind of dualist metaphysical argument about the possibility of AGI in the abstract. Merely a doubt that we can blindly scale up the complexity of a synthetic mind to meet or surpass our own. Consider that we still can't even begin to understand our own minds in enough detail to reliably predict, repair, or augment them. I am outside this field and so may have too much of a layman's perspective. But it seems to me that contemporary AGI believers conflate training and evolution. That nature did it in eons doesn't argue that we can do it in practical product development cycles, unless we can simulate these evolutionary processes to follow a similar search in a compressed time scale. As a crude analogy, I think todays LLM products are a bit like horoscope generators. Clever arrangements of words that attract a charitable or gullible reader. But AGI use cases are more like wanting a life partner who will understand and willingly assist ones efforts.