"The robot was best at Fine Arts and Current Events, worst at Word Play and Social Studies. ...This one’s not so surprising. We have a type of question called a

"The robot was best at Fine Arts and Current Events, worst at Word Play and Social Studies. ...This one’s not so surprising. We have a type of question called a “Two’fer Goofer” which asks for a pair of rhyming words that satisfy a given clue. It’s similar to the Rhyme Time category in Jeopardy or the old newspaper puzzle Wordy Gurdy. We had three of these questions in the showdown and GPT-3 missed all three of them. For Word Play questions that were more like vocabulary quizzes, GPT-3 performed admirably:" More evidence, if it was needed, that the use of BPEs sabotages GPT-3 in a lot of subtle ways (https://www.gwern.net/GPT-3#bpes https://www.gwern.net/GPT-3#bpes). GPT-3 can understand things like vocab definitions which do not depend on the internal spelling of a word or phonetics (which are erased by the BPE encoding of the data it was trained on), but as soon as you have to do things like puns... The BPEs are deadly. Ah well. Eventually OA or someone will train a proper character-level model, and then I'll be able to generate rhyming poetry without hacks like rhyming dictionaries.