"As a side note: I want to point out that I’m not aware of any examples of capabilities that can be done with prompting a model like GPT-4 that it can’t be trai

"As a side note: I want to point out that I’m not aware of any examples of capabilities that can be done with prompting a model like GPT-4 that it can’t be trained for. This is why I’m a little skeptical." While it is just a side-note in the article, isn't this the core of the problem? When we've already established that LLM's can do B to A generalizations in-context, why wouldn't they be able to in training? In one of my experiments I noticed that GPT-4 seems to be perfectly aware of the number of letters in a word in-context, but has difficulty when trained knowledge is involved. For example it can reliably answer the question: "Can you tell me how many letters each of the words of the first sentence of our conversation has?" At the same time it fails with the task: "Can you rewrite the first sentence of our conversation in a way that preserves its meaning as closely as possible but use only words with an even number of letters?" It will give an answer but get the letter counts very wrong and it is unable to improve its answer by iteration. Of course this does not prove that a model cannot be trained to answer tasks involving word length, but GPT-4 seems to have a knowledge gap here (possibly due to tokenization).