> Couldn't help but notice that 'Xanadu' uses a lot of rhymes ... GPT apparently didn't notice that constraint. Due to an obscure low-level technical decision

> Couldn't help but notice that 'Xanadu' uses a lot of rhymes ... GPT apparently didn't notice that constraint. Due to an obscure low-level technical decision (encoding text into BPEs), phonetics are erased from the GPT training data, and so therefore neither GPT-2 nor GPT-3 can learn to rhyme or make puns (they can only memorize relatively common pairs by bruteforce): https://www.gwern.net/GPT-3#bpes https://www.gwern.net/GPT-3#bpes I put a good deal of effort into testing this, including "Kubla Khan": https://www.gwern.net/GPT-3#samuel-taylor-coleridge https://www.gwern.net/GPT-3#samuel-taylor-coleridge Just doesn't work, and it never will until OA trains a model using some sort of better encoding which preserves phonetic information. (Character-based, unigram LM, or BPE dropout might all work, we don't know yet.)