2 ms·

Thanks - I think I understand you a bit better now. I suppose I was arguing that genetics is a simple system with emergent complexity (and that perhaps LLMs co

Thanks - I think I understand you a bit better now. I suppose I was arguing that genetics is a simple system with emergent complexity (and that perhaps LLMs could be too), but I suspect that betrays my limited understanding of genetics! > We don't train models on noise. We train them on coherent data. The very act of writing is where intelligence lies. A model can only present that intelligence. It cannot create it, and it is not aware of it. I think my main sticking point is the last sentence, which feels like a jump to me. For all the wisdom in your comments on complexity and invention, how does that link to awareness or understanding. Why does understanding require “ownership” of the complexity, so to speak? To me, that’s not clear; could these be emergent properties of a system where the complexity is generated elsewhere, for instance?

thomastjeffery 3y ago
Humans write. LLMs pretend to write. Really what they are doing is a statistical process called "continuation". By training an LLM on human-written content, we are essentially restricting the problem domain from "any possible next character" to "something similar to what the humans wrote". What the LLM does at the end of the day can be as simple and intelligent as blindly stumbling around whatever data it was given, because the domain it is stumbling around is made exclusively out of human-written content. An LLM won't generate nonsense, because nonsense lies outside its domain: the LLM was never trained on nonsense, so there is no nonsense for it to stumble into. An LLM can stumble into lies, offensive language, threats, incorrect arithmetic, logical fallacies, etc. These are generally presented as "limitations". The limitation is that we can't keep them out of the domain. It would be more clear to say those are limitations of the LLM creator. To the LLM itself, these limitations are a feature; though not the kind we are happy about. Humans are able to think about our writing before we write it. That's a feature called objectivity, and LLMs don't have it. There's no obvious place to add that feature, either.

thomastjeffery 3y ago

Humans write. LLMs pretend to write. Really what they are doing is a statistical process called "continuation". By training an LLM on human-written content, we are essentially restricting the problem domain from "any possible next character" to "something similar to what the humans wrote". What the LLM does at the end of the day can be as simple and intelligent as blindly stumbling around whatever data it was given, because the domain it is stumbling around is made exclusively out of human-written content. An LLM won't generate nonsense, because nonsense lies outside its domain: the LLM was never trained on nonsense, so there is no nonsense for it to stumble into. An LLM can stumble into lies, offensive language, threats, incorrect arithmetic, logical fallacies, etc. These are generally presented as "limitations". The limitation is that we can't keep them out of the domain. It would be more clear to say those are limitations of the LLM creator. To the LLM itself, these limitations are a feature; though not the kind we are happy about. Humans are able to think about our writing before we write it. That's a feature called objectivity, and LLMs don't have it. There's no obvious place to add that feature, either.