It sounds like you're suggesting that if it could get the number right, it would probably also be able to model people well enough to understand they would cons

It sounds like you're suggesting that if it could get the number right, it would probably also be able to model people well enough to understand they would consider it a mistake. We don't have to reason with counter-factuals here. GPT-3 just reproduces language. It isn't trained to be right, it's trained to predict and reproduce language so it has no incentive to be right. And the pattern of saying something correct but wound-sounding is quire rare in my experience. Most people who do that do it by accident as well I think. It also has no model of people and it knows no numbers. Everything it spits out is response to context, which does allow to seem to model people and seem to know numbers. No doubt, it has an association with some article describing the actual distance of an Everest ascent as well many other facts and misinformation about Everest. It will spit all this out in context.