What is the philosophical distinction between having "intention", and being a perfect behavioral simulation of something with intention? An LLM "merely" predict

What is the philosophical distinction between having "intention", and being a perfect behavioral simulation of something with intention? An LLM "merely" predicts the next token, but we do not interact directly with LLMs - we interact with carefully constructed characters which the LLM simulates. These characters can certainly have intention. Similarly, I think it's a mistake to say that "the output can only ever be average". Clearly that's not the case - ChatGPT already writes and answers questions to a well above average standard. The insight is that you can tailor the distribution from which your "average" is taken. Average what? Average web page? Average HN comment? Average essay by a once-in-century deep thinker? Crudely speaking, that is what "prompting" is all about. If all you ever got was "the average text", then you would have no control at all.