It is uncanny. I would expect a language model to pull up different person names for each time that one was called for. For a person name to be consistently us

It is uncanny. I would expect a language model to pull up different person names for each time that one was called for. For a person name to be consistently used through several paragraphs it is not enough to rely on word co-occurrence. If I had to produce a text like this I would simply take an existing text and replace randomly chosen words with other similar words (as hinted by amvalo). Similar as in - words that tend to occur in similar contexts. So John->Bob throughout entire text. But that would not be a language model product anymore, and where is fun in that? I should set aside some time to read this paper.