This seems like a "well, duh" type of result. But there's other factors at play. There's an evolution of LLMs happening right now, which is very similar to wha

This seems like a "well, duh" type of result. But there's other factors at play. There's an evolution of LLMs happening right now, which is very similar to what happened with stable diffusion. Specifically, there are publicly available LLM weights available. Many disparate, independent individuals and organizations are fine-tuning these models, creating LoRAs, and new foundational models. These weights are being merged, and further fine-tuned. This creates a system where humans are "evolving" the LLMs such that they change and adapt to requirements. And, of course, we select only for the best performing LLMs. Due to this, GPT-3 has been surpassed by open models, and we have specialist models that are better than GPT-4 in their domains. It seems to be only a matter of time that open LLMs are better than GPT-4. So, on their own, LLMs may not be able to generalize beyond their training set. But from a higher level, there is an ecosystem of evolving LLMs which do generalize (and sometimes specialize). As far as I'm aware, the fine tuning/creating LoRAs on new datasets that humans are doing could also be done in an automated fashion. So you'd need a system which accepts new external data/inputs, fine tunes on those, and some evaluation metric to choose the best LLMs. With such a system, it seems possible that LLMs could automatically learn and adapt to new information, much how humans do.