That's a No True Scotsman way of looking at language models. GPT-3 or any Transformer model doesn't need to be innovative to be successful. But that's why it's

That's a No True Scotsman way of looking at language models. GPT-3 or any Transformer model doesn't need to be innovative to be successful. But that's why it's important to run experiments and build test cases. In some cases, degeneration can actually be a feature: https://twitter.com/dril_gpt2 https://twitter.com/dril_gpt2