What OP is detecting is not a flaw in GPT-2 but in how GPT-2 is used to generate samples: it chooses only from the top k most likely words, so you can easily de

What OP is detecting is not a flaw in GPT-2 but in how GPT-2 is used to generate samples: it chooses only from the top k most likely words, so you can easily detect GPT-2 simply by running it and noting that no sampled word appears from below, say, the top-40 candidates, while human text will occasionally have a top-41 or a top-99 or a top-1000 word. If you fix that with a better sampling strategy like beam search or RL finetuning, unclear how well OP-like methods would work.