Agreed. And there's going to be some significant domain decisions for GPT-4 to consider: Will the made up words be existing single tokens? Will those single tok

Agreed. And there's going to be some significant domain decisions for GPT-4 to consider: Will the made up words be existing single tokens? Will those single tokens be heavily overloaded tokens, e.g. a letter of the alphabet? Will the representation of the circles be, say "B" for blue, or something else? Along with this are questions as to whether you're going to treat GPT-4 as a zero shot or multi-shot oracle; while they have this idea of 'context free' challenges in the paper, they, crucially, train their network first. Anyway, I like this paper less the more I think about it.