Article> One of their experiments, “Learning and Using Novel Words,“ strikes me as more remarkable than most of the others and the paper’s lack of focus on it c

Article> One of their experiments, “Learning and Using Novel Words,“ strikes me as more remarkable than most of the others and the paper’s lack of focus on it confuses me. This sort of "learning" is not necessarily real learning and it's not new for GPT-3. Even reduced GPT-2 willingly used made-up terms from the prompt in its results: https://medium.com/@VictorBanev/interrogating-gpt-2-345m-aaff8dcc516d https://medium.com/@VictorBanev/interrogating-gpt-2-345m-aaf... Search the article for 'Now I will feed it the same thing, but with a bunch of made-up terms.' It has some examples of how that stuff worked. I've already posted this in the original discussion of GPT-3 paper and I will post it again: statements about whether some system "learns new words" or "does math" require hypothesis formulation and testing. It astounds me that many people in ML community not only don't do these sort of things, but even actively oppose to the very idea of them being necessary. Recently there was a great live-stream from DarkHorse talking about this problem in science in general: https://www.youtube.com/watch?v=QvljruLDhxY https://www.youtube.com/watch?v=QvljruLDhxY They talk about "data-driven" science and the fundamental problems with that notion.