My first (almost) thought was "Ok, looking at the llama, where's the quality boost?" I think the explanation is probably in how LLM are trained. Even without kn

My first (almost) thought was "Ok, looking at the llama, where's the quality boost?" I think the explanation is probably in how LLM are trained. Even without knowing deeply the internals, I suspect that it's the compression of information so to simplify you can't make 8GB data contain all the facts of a "bigger" normalized relational database. So they keep the facts present everywhere and often drop rare facts. For example, a fact "SQL was invented at IBM", this fact can be found everywhere, in books, web sites, comments. You don't need access to copyrighted books to acquire this fact. But a first-person account of someone who worked at IBM at that time is probably can be found in a couple of books, but due to "compression", it will be gone anyway