How about a simple explanation of why it seems palpably absurd to some people that it will not be a dead end? Consider a teratoma or lab-grown organ. Is grafti

How about a simple explanation of why it seems palpably absurd to some people that it will not be a dead end? Consider a teratoma or lab-grown organ. Is grafting or engineering another attached sensory mechanism going to bring it closer to being an ordinary organism? I think whether capabilities are super- or sub- human is a red herring. Even a really primitive organism is still taking all of its computational capabilities and outputting, implicitly, decisions in one context that is its perceived reality. A collection of computation and perception modules does not do this, without something else. I don't think developing "something else" is obviously impossible or would require magic. But I'm not sure anyone sane would want to create it when it inherently creates unlimited risk of running amok. This is what the LessWrong people are afraid of, aren't they?