You're definitely onto something here. I look at it from a realist take - it's just language. Language can do impressive things and have impressive consequences

You're definitely onto something here. I look at it from a realist take - it's just language. Language can do impressive things and have impressive consequences, but they aren't sticks and stones. We ascribe high value to well-written text because it's seen as this token of intelligence, but on the other side of the board it's a game of pattern-recognition. We may well impress ourselves by how much work can be saved with guessing the rest of your response. As you say, the original sin is anthropomorphizing AI. We all want to do it, because it's a convenient way of looking at it and extremely gratifying to our inner science-fiction nerd. It has the unfortunate side-effect of implying something has a soul, or a lived experience that is the sum of something greater than text. It's not. Even AI trained on text and images and video is not capable of simulating a human life. The only way to accurately model the success, failure, longing and achievement of humankind is by growing into it one year at a time. Claims of "superhuman" writing speeds may well be true, and the text might even be written inscrutably well. It's not human, though, and humanity is a poor benchmark for LLM scientists that ought to know better.