The bottom line is that you're demanding that LLMs somehow magically get to a certain level of functionality, without using input that you are allowing humans t

The bottom line is that you're demanding that LLMs somehow magically get to a certain level of functionality, without using input that you are allowing humans to have before you evaluate their functionality. How that input is used isn't relevant. Humans don't have separate training and inference training phases, but that does not mean that a human doesn't have to learn from other humans. Including just plain ingesting streams of stuff without interaction, although that's not the dominant mode for humans. By the way, any LLM you have ever interacted with has undergone reinforcement post-training in which it did in fact converse. The results of that are baked into the weights, not in the context. Not that it matters for the unreasonableness of your "test". And I predict that within a couple of years the training-versus-inference distinction in LLMs will start to break down. Which still doesn't matter when you're evaluating their current performance.