I think you're right. This is discussed in the article- GPT-3 does very well at the small-scale, like short poems. The architecture of GPT-3 is based on Word

I think you're right. This is discussed in the article- GPT-3 does very well at the small-scale, like short poems. The architecture of GPT-3 is based on Word2Vec, and attention - it doesn't really have a good understanding of large-scale semantic structures in writing. (Also mentioned in the article - it can't rhyme: it doesn't understand sub-word semantic structure either). I think the next big advance will be when we are able to apply attention/word2vec-style algorithms at other semantic levels.