Of course language relates to things, but I am thinking of language in the sense of a constantly growing corpus of text. It is a self replicator, but like a vir

Of course language relates to things, but I am thinking of language in the sense of a constantly growing corpus of text. It is a self replicator, but like a virus it uses the host organism, a human or a LLM to create more of itself. Ideas have their own life-cycle, they don't get born or die with us, they like to travel and evolve fast. My point was the capabilities we attribute to LLMs actually come from the training data, not the model. Any model would do, from 10B parameters to 500B, with cosine embeds, rotary or AliBi, even RNNs can do it today. But the data - that is precious, the source of all these "emergent" abilities. We are wondering at the mysteries of the transformer, but the mystery was hiding the training set. Let's look at the problem from the point of view of language as an evolutionary system of ideas. Like scientific papers and citations, that kind of stuff, leading to AI and human progress. That's where our progress will come from - the evolution of ideas.